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Evaluating Your Program is a resource to help program leaders 
evaluate their programs using fidelity and outcome assessment 
tools. These evaluations may be done as self-assessments or as 
external evaluations. 

Self-assessments are conducted by the program leadership 
and membership itself for various purposes such as program 
development or quality improvement. External evaluations are 
conducted by people outside the program, often professional 
evaluators from universities or government. The purpose of 
external evaluations may be for research or to demonstrate that 
the program is achieving desired outcomes or meeting evidence 
based practice (EBP) standards. 

Evaluating Your Program offers guidance for both internal 
and external evaluators in preparing for and conducting 
fidelity assessments and outcome evaluations for consumer- 
operated services. 

The Fidelity Assessment Common Ingredients Tool (FACIT) is 
highlighted and the FACIT protocol, scoresheet, and an Excel 
Workbook User's Guide are included in the appendices. The Peer 
Outcomes Protocol (POP) is also discussed as an instrument for 
outcome evaluation with information on how to access this resource. 
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For references, see the booklet, The Evidence. 
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Preparing for an Assessment or Evaluation 


This section provides basic information and tips for preparing for any kind of 
program assessment or evaluation. It includes information about why and how 
to conduct an assessment or evaluation, common concerns, and measurement tools. 
It also differentiates between a fidelity assessment and an outcome evaluation. 


Why conduct a program 
assessment or evaluation? 

How is your consumer-operated service 
performing in relationship to its goals? 
What peer practices are staff members 
following to meet the needs of program 
participants or members? How is your 
consumer-operated service affecting 
participants’ efforts to recover? 


An organizational evaluation helps to 
answer these and other questions that 
affect the quality of the programs that 
your consumer-operated service delivers. 
The evaluation also will enable your 
organization to meet the growing call for 
accountability from federal, state, local, 
and private funding sources. 
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It is clear that as states adopt a recovery-based 
approach to mental health by delivering evidence- 
based practices, consumer-operated services need 
to conduct evaluations to examine the fidelity and 
effectiveness of the peer activities they provide. 
However, the call for accountability from funders 
and for responsiveness to participants poses 
challenges for program leaders. 

In a national survey of data needs of peer support 
programs, Campbell (1997a) found that almost 40 
percent of the programs surveyed felt that collection 
of member information would discourage people 
from using the services, and 43 percent felt that 
it would financially burden the organization. 

On the otlier hand, evaluation has been instrumental 
in helping consumer-operated services respond to 
growing demands for evidence-based practices and 
has provided these organizations with opportunities 
to deliver peer services more effectively. 

The capacity of program leaders to monitor and 
improve program quality and outcomes has been 
established through recent nationwide initiatives. 
These projects give the program the scientific tools 
required to assess and tailor services so they better 
meet the needs of individuals with mental illnesses. 
The Consumer-Operated Services Program (COSP) 
Multisite Research Initiative is among the most 
notable nationwide efforts successfully completed 
by peer providers. 


A growing number of consumer-operated 
program leaders have learned basic evaluation 
skills (Campbell, 1997b), and the value of including 
consumers in the process has been demonstrated. 
The validity of survey tools may be improved when 
consumers serve as investigators, coordinators, 
surveyors, data managers, analysts, or focus group 
members. In addition, improvements may occur in 
the quality of the following: 

■ The interview process; 

■ Data interpretation; and 

■ Dissemination strategies. 

Recent studies have found that consumer-operated 
programs have reduced expenses, generated income, 
increased efficiency, and increased service demand 
by implementing quality improvement practices 
that incorporate evaluation findings (Campbell 
and Leaver, 2003). 

In addition, consumers who participate in evaluation 
projects experience both economic and personal 
benefits, including the opportunity to help other 
consumers, increase communication and job- 
related skills, and enhance their self-esteem. 

One of the first steps in conducting a fidelity 
assessment or outcome evaluation is to decide 
why you want to do it. For example, 

■ Is it required by your funder? 

■ Are you interested in program 
quality improvement? 

■ Do you want to demonstrate and document 
success to enhance future funding? 

The purpose and goals shape the methods and 
tools used to conduct the assessment or evaluation. 
Make sure that everyone involved—including any 
outside evaluators—understands and agrees with 
the identified purposes and goals. 
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Defining fidelity assessment 
and outcome evaluation 

Fidelity assessments and outcome evaluations 
are not the same thing. This section distinguishes 
between them. 


About fidelity assessments 

Fidelity assessments measure whether a program 
operates according to standardized practices. 

Fidelity assessments use objective rating systems 
(also called tools, measures, or instruments) with 
instructive protocols to determine the extent to 
which program components adhere to established 
models, theories, or philosophies. 

The fidelity measures developed for consumer- 
operated services enable program leaders to 
pinpoint program departures from standard 
practices and models (Flolter, Mowbray, Bellamy, 
MacFarlane & Dukarski, 2004; Johnsen, Teague 
& McDonel-Flerr, 2005). Research findings 
to date indicate that fidelity assessments should be 
used to inform strategic planning, staff training, 
program development, and other quality 
improvement initiatives. 


About outcome evaluations 

Outcome evaluations measure whether a program 
achieves desired results for participants. 

While fidelity assessments look at how services 
are provided, outcome evaluations measure 
program successes and consumer satisfaction 
and achievements. Some outcomes are concrete 
and observable, such as the number of days 
a consumer has been employed or has remained 
in independent housing. Others are more 
subjective and personal, such as gaining a sense 
of empowerment or being satisfied with services. 


It is important to measure outcomes as well as 
fidelity Good fidelity alone does not necessarily 
guarantee effectiveness and poor fidelity does not 
prohibit achieving desired outcomes. In addition, 
fidelity results cannot identify groups of people 
for whom services are effective. 

Concerns about program assessment 
and evaluation 

Although the value of assessments and evaluations 
has been demonstrated through research and 
field experiences, the benefits may not be readily 
apparent. In addition, conducting these studies 
may create financial or administrative challenges. 


Common concerns 

Some consumer-operated service program leaders 
and participants are skeptical about implementing 
evidence-based practices (Essock et ak, 2003). 
Consumers ask “whose evidence?” and “evidence 
of what?” They point out that measures of fidefity 
to standardized practices may provide results that 
are antithetical to the consumer movement’s goals 
or belief systems. 

For example, some consumers resist efforts to collect 
program and member data, believing these activities 
destroy the program’s ability to provide participants 
with an informal, safe, noncoercive environment. 

Some peer providers also are concerned by the 
desire of mental health authorities or other funding 
agencies for a comparative framework using an 
objective system such as FACIT to evaluate the 
performance of consumer-operated services. 
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Financial and administrative challenges 

Smaller consumer-operated services may face 
significant challenges in implementing fidelity 
and outcome studies, For example: 

■ Staff may lack the necessary knowledge and 
skills; and 

■ Costs for fidelity assessments and outcome 
evaluations and for implementing and 
monitoring the results (including personnel, 
equipment, and software) may be prohibitive. 


Addressing these challenges 

Addressing these challenges requires identifying 
opinion leaders among members, staff, and 
advisors and communicating with them clearly, 
honestly, and calmly to build support for the 
assessment and/or evaluation. 

It also requires flexibility in responding 
to financial, organizational, and community 
concerns. For example, peer providers should 
use basic assessment methods until the service 
has reached organizational capacity to take on 
more complex approaches. 

You also will need to take an inventory of 
what the organization has and what it needs 
to successfully conduct analyses and implement 
evidence-based practices. 


Who can conduct an evaluation? 

When a program decides to conduct an evaluation, 
it must first decide whether it will be done as a 
self-assessment or whetlier outside evaluators will 
be hired. Minimally, a two-person team should 
conduct the evaluation, but there can be more 
people on the team. A team provides multiple sets 
of ears to make sure things are heard accurately 
and many hands to help with note taking and other 
tasks. Regardless of the choice of “who,” it is 
important that the evaluators fully understand the 
goals, questions, and procedures of the evaluation. 

It is also important to avoid potentially biased 
evaluators who are invested in the program 
“looking good” (or bad). Evaluators should not 
have a conflict of interest such as being on the 
organization s board of directors or working as 
the program leader. Such dual responsibilities 
put the evaluators in a compromised position 
about the infonnation learned about the program, 
staff, and members. If the evaluation is to be 
done as a self-assessment, it is ideal to establish 
a Continuous Quality Improvement (CQI) team 
made up of members to lead the evaluation. 
Neighboring programs could also exchange 
evaluators to reduce conflicts of interest. 
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Which measurement tools to use? 

Once you decide to proceed with a program 
assessment or evaluation, you will need to make 
choices about which measurement tools to use 
and who should conduct the study. 

A program may elect to use any of a variety of tools 
for fidelity assessments or outcome evaluations. 
However, a fidelity assessment tool, the FACIT, 
and an outcome evaluation tool, the Peer Outcomes 
Protocol, have been developed to respond to the 
informational needs of consumer-operated services. 

This section provides more information about 
these two measurement tools. 


The Fidelity Assessment Common 
Ingredients Tool (FACIT) 

FACIT incorporates key components of the peer 
support and self-help philosophy. The tool: 

■ Was developed by leaders in consumer- 
operated services; 

■ Is based on program choice; and 

■ Is intended to build skills and to empower users. 

A copy of the FACIT can be found in the appendices 
of tliis booklet, along witli a Users’ Guide. 

FACIT's history and success 

FACIT has a relatively short history, but programs 
throughout the United States have begun to adopt 
the scale. FACIT was developed as part of the 
Consumer-Operated Services Programs Multisite 
Research Initiative (1998-2006). This collaborative 
effort enabled peer providers and researchers to 
(I) identify a set of standard elements common to 
consumer-operated services and (2) develop a tool 
to measure the extent to which programs used 
these common elements or ingredients. 


As part of the initiative, members of consumer- 
operated services collaborated with researchers 
to generate a list of common peer-run program 
ingredients (Johnsen, Teague, & McDonel-Herr, 
2005). The ingredients were developed from a 
comprehensive review of peer literature that 
generated more than 170 items and a national 
“concept-mapping” effort in which program 
leaders ranked and sorted the items to identify 
peer support competencies (Campbell, Dumont, 

& Einsphar, 1999). 

The list of common ingredients was tested in two 
rounds of multisite visits. During the first round, 
conducted from 1999 through 2001, researchers 
and consumer-providers pilot-tested FACIT and 
determined that it successfully measured common 
program ingredients. In the second series of 
visits, FACIT was successfully used to measure 
the presence of the common ingredients 
in eight programs. 

FACIT has undergone extensive psychometric 
testing and has demonstrated discriminant validity, 
meaning that it can be used to differentiate between 
peer-run programs and traditional mental health 
services. The tool also has demonstrated predictive 
validity, which means that it can be used to tie 
scores to consumer outcomes, particularly those 
related to recovery and well-being. These outcomes 
include hope, empowerment, meaning of life, 
self-esteem, and social support as measured in the 
subscales of the Composite Well-Being Measure 
(Campbell et ah, 2006). 
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Components of a FACIT assessment 

Collecting information from multiple sources helps 
evaluators more accurately capture how services 
are provided and identify the common ingredients 
incorporated in the program. FACIT components 
include the following: 

■ An interview with the program director; 

■ Focus groups conducted with staff and program 
members or participants; 

■ Observations of daily program activities, (including 
program groups, provision of services by staff, 
and informal participant interactions); and 

■ A review of program materials. 

Smaller programs may not have the number 
of staff needed for a focus group. In these cases, 
staff may be interviewed individually. 


The Peer Outcomes Protocol (POP) 

As the number of peer-run programs continued 
to grow during the past two decades, so did the 
demand from program leaders for an outcome 
assessment with measures derived from mental 
health consumers’ experiences and points of view. 
In response, the POP Project was initiated. 

POP's history and success 

The POP was largely designed, tested, and 
implemented by consumer researchers, advocates, 
and program leaders who had mental illnesses. 
Direction was provided by Dr. Jean Campbell 
of the Program in Consumer Studies and Training 
at the Missouri Institute of Mental Health. 

The POP Project was developed between 1995 
and 2000 and funded by the University of Illinois 
at Chicago’s National Research and Training 
Center on Psychiatric Disability. The POP has 
been successfully used by consumer-operated 
programs throughout the United States. 

More information and copies of the POP 
instrument can be found at the POP Web 
site: http://www.cmhsrp.uic.edu/nrtc/pophome.htm. 
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Conducting a FACIT Fidelity Assessment 

This section provides information for organizing, preparing for, and conducting 
a FACIT fidelity assessment. It discusses the FACIT Team, preliminary organizational 
inventory, organizing the assessment process, conducting and scoring the assessment. 

The appendices to this booklet contain the FACIT protocol, scoresheet, 

and an Excel Workbook User's Guide designed to tabulate FACIT data over time. 


The FACIT Team 

This section provides information to help 
you select evaluators for conducting a 
FACIT assessment. It identifies key 
evaluator responsibilities and provides 
tips for selecting individuals to serve 
on the FACIT Team. 


Key FACIT Team responsibilities 

The FACIT Team members must conduct 
interviews and focus groups, observe 
program activities, collect the required 
data, and analyze the results in ways 
that will help programs improve their 
fidelity to evidence-based practices. 
Furthermore, the team must ensure that 
the information is collected with respect 
for the confidentiality of the respondents 
and the data and recorded carefully, with 
provisions made for ensuring its accuracy. 
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FACIT Team members should have experience 
as interviewers or focus group leaders. They also 
should be comfortable using the data collection 
and analysis software available to the program. 

In addition, they should be culturally aware 
and have the competencies needed to conduct 
successful interviews and focus groups with 
members of the major demographic/racial/ethnic 
groups involved in the consumer-operated program. 

Selecting the FACIT Team 

Two evaluators are usually selected to work 
together as the FACIT Team. They may be either 
independent consultants familiar with the program 
and the consumer support philosophy or program 
staff, members, and advisors. 

Working with an independent FACIT Team may 
be the better option if these evaluators are truly 
objective and competent. Options for finding 
relatively low-cost external evaluators include 
the following; 

■ Arranging with other neighboring consumer- 
operated programs to exchange evaluators; 

■ Hiring an outside evaluation team from a local 
community or an academic institution; and 

■ Asking the local mental health authority 
to volunteer staff to conduct the 
FACIT assessment. 

Because of financial circumstances, however, 
in-house evaluators often are selected. They 
can conduct fidelity assessments in a capable, 
competent manner. Members of the advisory 
committee or board of directors may be trained 
and may rotate responsibility for completing 
fidelity assessments. Alternatively, if the organization 
has a quality improvement team, its members may 
be designated to undertake the FACIT processes. 


Taking inventory 

In addition to selecting the assessment tools 
and evaluators, you should determine what the 
organization needs to get from the assessment. 

This requires inventorying the program’s strengths 
and weaknesses. 

It can feel daunting to think about inventorying 
organizational strengths and weaknesses in 
preparation for conducting the assessment. However, 
the Readiness Assessment Checklists provided on 
the next pages may provide a useful structure and 
a set of helpful prompts for conducting this inventory. 

Readiness Assessment involves two 
separate checklists: 

■ General Knowledge; and 

■ Program Structure, Personnel, Resources. 

The Readiness Assessment Checklists and 
instructions for using them are found on the 
following pages. 


Tips for Selecting Evaluators 


Peer providers should— 

■ Select individuals not centrally involved 
in providing program services. 

■ Be wary of the potential biases of 
evaluators who are invested in seeing 
the program "look good" or who do not 
fully understand the ingredients common 
to consumer-operated services. 

■ Select evaluators who can work together 
as a team. 

■ Make evaluation personnel decisions early 
in the planning stages of the evaluation. 
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What do you need for data collection 
and analysis? 

The methods you use for data collection depend 
on the data you want to collect. Once you have 
agreed-upon outcomes, identify the methods 
you have available for collecting the information 
you need. These methods may be as simple as 
using and collecting activity sign-in sheets and 
maintaining an updated list of program members. 

Consult with prospective evaluators and other 
consumer-operated programs about other efficient 
data collection methods. Frequently used data 
collection methods include surveys, focus groups, 
and interviews. 

Determine your program’s software and 
hardware needs by reviewing in-house 
capabilities and resources identified by 
the Readiness Assessment Checklist. 

Consult with prospective evaluators and staff 
from other consumer-operated programs who have 
conducted assessments. Required equipment and 
software commonly include: computer capacity, 
spreadsheet software (e.g., Microsoft Excel), basic 
office supplies, and a telephone. 


Other actions 

In addition to reviewing items identified in the 
checklists, you should thoroughly examine tlie 
program’s mission and gather feedback from 
members about where improvements are needed. 

After this inventory is completed and possible 
evaluators are identified, you are ready to 
undertake the specific preparations discussed 
in the next section. 
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Part 1: General Knowledge 

Part 1 of the Readiness Assessment Checklist, General Knowledge, is structured to help leaders, staff, 
and members identify topics about which they want to learn more before they proceed. At the bottom 
of the Part 1 checklist, note topics where you need more information. Please refer to the resources in 
The Evidence booklet included in this KIT for sites and sources to help you find this information. Consider 
also consulting with state and local evaluation experts at area universities and at the consumer technical 
assistance centers. 

Check any areas that you feel you do NOT understand. Cross out items that do not apply to your organization. 

At the end of the checklist, note other areas that were not listed where you have questions, and list what 
steps you might take to address these questions. 


□ The purpose of consumer-operated services 

□ The history of the consumer movement 
and peer support services 

□ The meaning of “consumer-operated” 
and “member-run” 

□ The meaning of the “recovery” concept 

□ How your program’s fidelity to the program 
model will be measured 

□ How the system for collecting outcome 
data will work 

□ Roles of staff/members/volunteers 
at a consumer-operated program 

□ Leadership development 

□ How to motivate your staff in implementing 
FACIT peer practices 

□ Administrative tasks 

(e.g., budget, personnel, supervision) 

□ Specific admission criteria for your program 

□ How to conduct staff and membership meetings 

□ The relationship between the program and 
advisory groups and/or the board of directors 

□ How to facilitate peer support groups 

□ Techniques for writing proposals or reports 


□ How to develop a marketing plan or conduct 
community outreach 

□ How to establish a collaborative relationship 
with traditional mental health providers 

□ Methods for preparing a program manual 
NOTES 
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Part 2 of the Readiness Assessment Checklist, Program Structure, Personnel and Resources, includes practical 
matters that should be considered; these are grouped into seven subject areas. The lists are offered as 
general tools to help your consumer-operated service reach its own goals. Use this list to identify elements 
that are in place in your organization and those that need further development. 

Check items that have been completed. Cross out items that do not apply to your organization. 

Note the next steps you may want to take to obtain or complete the items you did not check. 


2.1 Staffing and Membership 

The organization has the following: 

□ A director 

□ A program coordinator 

□ A bookkeeper or outsourced 
bookkeeping services 

□ Support group facilitators 

□ Quality improvement team 

□ A mission statement 

□ Written job descriptions 

□ Staff training programs 

□ Volunteer recruitment and training programs 

□ Staff reflecting the cultural diversity of the 
local community 

□ Empowerment of staff/membership in 
decision-making 

□ Written membership criteria 

□ Posted membership rules 

There are written personnel policies concerning 
the following: 

□ Maintaining confidentiality 

□ Prohibiting sexual harassment 

□ Complaints and grievances (staff/members) 

□ Maintaining a substance-free environment 

NOTES 


2.2 Program Budget 

The organization has funding for the following 
services and items: 

□ Competitive salaries and fringe benefits 

□ Rent, utilities, and facility maintenance 

□ Required types of insurance 

□ Telephone and communication equipment 
(e.g, pagers, cell phones, Internet access, 
postage meter) 

□ Office supplies (e.g., pens, paper, toner) 

Office equipment, including the following: 

□ Computer(s) 

□ Fax machine(s) 

□ Copier(s) 

□ Printer(s) 

Office furniture, including the following: 

□ Storage and file cabinets 

□ Desks 

□ Chairs 

□ Table(s) 

Travel and transportation resources, 
such as the following: 

□ Vehicle lease/purchase 

□ Travel reimbursement 

□ Rus passes 
NOTES 
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2.3 Facility and Equipment 

The organization — 

□ Is situated in a safe location that is convenient 
for the community it serves. 

□ Has the space, equipment, and technology 
needed to offer programs in a consumer-friendly 
environment and to enable staff to carry out 
their duties efficiently 

□ Provides reasonable access for people 
with disabilities. 

NOTES 

2.4 Program Resources and Materials 

□ Volunteer recruitment and training programs 

□ Computer/Intemet access for members 

□ Peer support and wellness literature 

□ A calendar of activities 

□ A newsletter or Webpage 

□ Display(s) of motivational/artistic/recovery- 
focused materials 

□ A community resource list 

□ Supplemental resources, such as the following: 

□ Videos/DVDs/computer games 

□ Art materials 

□ A small refrigerator and microwave 
or kitchenette 

NOTES 

2.5 Data Collection and Quality Improvement 

The organization has these resources: 

□ Computer/Intemet access for members 

□ A system in place for capturing and 
analyzing data 


□ A process for providing information to key 
stakeholders and incorporating their responses 

□ Mechanisms for tracking followup activities 
NOTES 


2.6 Services 

The organization provides the following: 

□ Fonnal peer support groups 

□ Recreation 

□ Greeter/new member orientation 

□ Informal peer-to-peer support 

□ Wellness management 

□ Skills training 

□ Job readiness 

□ Meals/snacks 

□ Advocacy 

NOTES 


2.7 Program Marketing and Outreach 

The organization has the following marketing/ 

outreach tools: 

□ Brochure(s) 

□ Flyers/posters 

□ Access to free advertising in media outlets 
(e.g., newspaper, cable television, public radio) 

□ Representatives on community provider boards 
and committees 

□ Representatives participating in 
community initiatives 

□ A referral system 
NOTES 
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Preparations for a 
FAC IT assessment 


Sample FACIT Agenda 


Next steps include appointing an individual 
responsible for coordinating the assessment 
and identifying key responsibilities to be shared 
between the two of you and those to be taken 
by the FACIT Team. Responsibilities include 
gathering key materials, promoting recruitment, 
and ensuring that confidentiality and other 
requirements are met throughout the assessment. 

Assign an assessment coordinator 

If working with an outside evaluator, it is important 
to have one key person at the consumer-operated 
program as an assessment coordinator or liaison. 
The assessment coordinator links the evaluators 
and the program, communicates its purpose and 
scope to program staff and participants, and 
provides logistics such as schedufing and gathering 
required materials. If the organization has a quality 
improvement team, the coordinator should be a 
member of this group. 

Create a schedule 

One of the coordinator’s main responsibilities is 
developing a schedule that includes the required 
interviews, focus groups, and site observations. 

This should be developed well in advance of the 
site visit to respect competing time demands on 
the program director, staff, evaluators, 
and participants. 

A 2-day site visit is optimal for conducting the 
assessment. Each interview or focus group session 
should run between 90 minutes and 2 hours, 
with time allowed for refreshments. The text 
box=:provides a sample agenda for a site visit. 


Day One 

Morning 



8:00-8:30 

Review agenda 


8:30-11:30 

Interviews/focus 
groups for staff 


Afternoon 



Noon-2:00 

Lunch and facility tour, 
observation of activities 


2:30-4:00 

Interview program director 

Day Two 

Morning 



9:00-11:00 

Conduct focus group with 
program members 


11:00-noon 

Review program materials 


Afternoon 



Noon-1:00 

Lunch 


1:00-2:00 

Observe peer support 
group and other activities 


2:00-3:00 

Conduct exit 
interview (optional) 


Gather key documents 

The efficiency of the assessment is enhanced when 
program materials and documents are gathered 
beforehand. This involves collecting tlie information 
identified in the FACIT Materials Checklist and 
preparing a packet of these materials for each 
evaluator. The FACIT Team will review these 
materials with the director during the site visit 
and will use them in the FACIT scoring. 

The text box provides a FACIT Materials Checklist 
that outlines the key documents to be prepared for 
the FACIT Team. 
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FACIT Materials Checklist 


□ Articles of incorporation 

□ Agency mission statement/philosophy 

□ Organizational structure/organizational 
chart 

□ Bylaws 

□ Advisory council/board minutes from the 
last 12 months 

□ Program operating budget 

□ Policy and procedures manual 

□ Sexual harassment and grievance policies 

□ Satisfaction survey 

□ Participant demographics (if collected) 

□ Unduplicated monthly attendance record 

□ Reports to funders or annual reports 

□ Activity calendar/newsletter 

□ Training manuals 

□ Job descriptions 

□ Brochures 

□ Letters or advertisements 
soliciting volunteers 


Recruit members for focus groups 

The program leader, FACIT Team, and assessment 
coordinator may assist with recruiting program 
members for focus groups. The program leader 
should ensure that recruitment is initiated and 
invite staff and members to participate in 
interviews or focus groups as appropriate. 

Between 15 and 20 individuals should be recruited 
to participate in the member focus group. 

To minimize selection bias, the recruited 
consumers should be a cross-section of the 
individuals who typically attend the program. 

One method for recruiting program members is 
to simply post a flyer at the program inviting them 
to participate. In addition, the event may be noted 
in advance on the organization s calendar of events. 
To further promote participation, the FACIT 
process might be discussed at a member meeting. 

Participants also might be recruited using random 
selection. In this process, the names of all program 
members are put in a hat and selected through 
a chance drawing. Those selected are tlien invited 
to attend. If not enough people agree to 
participate, a second drawing is conducted. 
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Maintain participant confidentiality 

Program members, staff, and the FACIT Team 
should ensure that confidentiality procedures are 
followed. Ultimately, however, the program leader 
is responsible. Before the assessment begins, he 
or she should carefully review the confidentiality 
procedures with all of the involved parties. 

Points to consider include the following: 

■ Participation in interview/focus groups is voluntary. 
However, a written informed consent form is not 
required when data are used solely for quality 
improvement purposes rather than for research. 

■ A breach in confidentiality could arise during 
a focus group if a person divulges private 
information about his or her mental health 
experiences to other participants. Because all 
focus group participants are program members, 
by definition they are self-identified consumers 
of mental health services. Therefore, little 

or no harm should result if a group member 
discloses his or her status as a person with 
a mental illness. 

However, there should be a discussion about 
the necessity for participants to not repeat 
anything heard in the focus group. Beminding 
participants that confidentiality protections in 
a focus group are the same as those protections 
in a peer support group should build 
understanding and enhance cooperation. 

■ Each interview or focus group must be held 
in a private setting. 

■ FACIT Team members must be cautioned 
not to share anything that is said during an 
interview or focus group session with other 
program members. 

■ No one other than a participant or FACIT 
Team member should sit in on interviews 
or a focus group. 


Maintain confidentiality of data 

All data collected from the assessment activities 
are confidential. The program leader and FACIT 
Team should take great care in explaining this to 
participants and in following procedures to ensure 
that data are kept confidential. 

The program leader has responsibility for data 
confidentiality when the information is onsite. 

If the FACIT Team takes the data, the team 
is responsible for ensuring confidentiality. 

Assessment data, such as FACIT scoresheets and 
interview/focus group notes and audiotapes, are to 
be stored in a locked file cabinet. This information 
should be kept for about 3 years; it may be needed 
to help address any data quality questions or for 
use in possible re-analyses. 
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FAC IT Team tasks 

The following tips about preparing for the FACIT 
process are pertinent for, and directed to, both 
internal and external evaluators. All FACIT 
Teams conduct similar activities and have similar 
responsibilities. These include conducting 
interviews and focus groups, observing regular 
program activities, collecting and analyzing the 
data, and developing a report assessing how well 
the program follows evidence-based practices. 


The best environment for conducting an 
assessment is created when everyone involved 
in the process wants to learn the facts and use 
them to improve their consumer-operated program. 


Establish a shared understanding 
of assessment goals 

After discussion with the program leader, 
communicate the assessment goals to program 
staff and members. Keep in mind that the most 
successful assessments are those in which all 
parties agree with the following; 

■ They want to understand the program s strengths 
and weaknesses so they can plan for its ongoing 
success; and 

■ This can best be done using standardized, 
scientifically established measures. 

If program staff or members fear that the program 
will lose funding or “look bad” if it does not 
score well, then the accuracy of the data may 
be compromised. 


Clarify reporting procedures 

The FACIT Team should clarify who will receive 
a copy of the fidelity assessment results and any 
reports that are prepared. Report recipients may 
include administrators, staff, and/or participants in 
the consumer-operated service as well as members 
of the board of directors or advisory committee. 
The team also should determine whether the 
report will be sent via e-mail, fax, or surface mail. 

Review, tailor, and organize 
assessment materials 

The FACIT Team should review and prepare 
specific materials for the assessment. This includes 
the following: 

■ Reviewing the FACIT materials in Appendices 
A and B and tailoring (as needed) key forms 
for use in specific assessments; and 

■ Developing an understanding of the history, 
operations, and beliefs of consumer-operated 
programs. To this end, FACIT Team members 
may benefit from reading On Our Own 
Together: Peer Programs for People with 
Mental Illness (Clay, 2005), a book on 
developing peer-run programs that addresses 
the philosophy and history of the movement 
and the tools needed for success. 

□ Each team member should prepare a 
binder containing program information 
and assessment tools. The contents should 
include program contact information and 
directions, the agenda, and tailored sets of 
questions and prompts for the interviews and 
focus groups. The binder also should include 
the FACIT scoresheets, sample program 
materials, and other items that will be of 
use such as name tags for the evaluators 
and focus group participants. 
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Other preparations 

Other preparations include the following: 

■ Ensuring that recording equipment is working 
and that related supplies, such as extra batteries, 
are included in the equipment brought to 

the program; 

■ Contacting the program leader and/or 
assessment coordinator before the site visit to 
ensure that (1) the schedule is fully understood 
by all parties and provides adequate time for key 
activities and (2) appropriate space has been 
reserved for conducting interviews and focus 
groups; and 

■ Talking with the program leader about whether 
an exit interview will be conducted. 


The FACIT Team: 

Scoring the assessment 

This section provides guidance to help 
FACIT Teams correctly score the FACIT data 
obtained from the interviews, focus groups, and 
program observations. 

FACIT Team debriefing and followup 

After the focus group concludes, the FACIT Team 
should briefly write out short observations of the 
event. Characteristics of the physical environment 
that helped or hindered the process need to be 
noted. Describe the way the group 
dynamics unfolded. 

For example, did leaders emerge? How was the 
flow? What did the comfort level appear to be? 
Also document what worked and what seemed to 
block the interview/focus group processes. 

The team also should follow up on any missing 
data, beginning by contacting the program leader. 

FACIT scoring 

The team should review the FACIT protocol 
{Appendix A), scoring sheet {Appendix B), 
and FACIT Excel Workbook User’s Guide 
{Appendix C). 

Each member should independently review the 
data collected, rate the scales, and tally the item 
scores. Team members should then compare their 
ratings and resolve any disagreements using the 
conciliation process and record the consensus rating. 
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As part of the scoring process, the team should 
carefully evaluate the relevance and quality of 
the evidence collected for each item. Evidence 
includes the following: 

■ Comments made by staff and members during 
interviews and focus groups; 

■ Observations of environment and interactions; 

■ Materials available to program members 

(e.g., newsletters, calendars, brochures, pictures 
of member activities, member artwork); 

■ Organizational policies and procedures; 

■ Staff training manual; 

■ Board and member meeting minutes; 

■ Informal discussions; 

■ Previous evaluations; and 

■ Satisfaction and outcome reports. 

A FACIT Team member needs to engage 
in “detective work” of a sort, following up 
on comments gathered through interviews 
or focus groups and collecting documentation 
to substantiate assertions. This includes comparing 
various pieces of evidence to determine the 
optimal item scores. 


The highest rating for each item should be 
reserved for programs that clearly meet the 
criteria for that rating. In no case should a partial 
score or fraction of a rating point be assigned. 


For example, if staff says that the program has a 
strong arts component, the evaluator should look at 
the studio, art materials, and participant artwork to 
get a better sense of participants’ opportunities in 
this area. In general, evaluators will obtain a more 
accurate assessment by using the comparative 
approach than from relying on a single source. 


Pointers on how to resolve possible problems with 
interpreting data include tlie following: 

■ Rate the program based only on currently 
implemented activities and practices. Do not 
score a program based on past or future 
activities and practices. 

■ If members differ considerably in their 
responses to the same question, look for other 
data. For example, where one respondent 
provides an answer but expresses uncertainty 
about it while another provides a different 
answer but is quite certain about it, look 

for other documentation, such as newsletter 
articles, calendar listings, and flyers. If there 
is any question about program activities 
and practices that cannot be verified by 
the evidence, call or e-mail program staff 
or members for further clarification. 

Initial rating 

To complete the initial ratings, each FACIT Team 
member should independently complete the FACIT 
scoresheet. This involves the following steps: 

■ Assemble all relevant information. This includes 
interview and field notes, documentation 
provided prior to the FACIT assessment, 
other documentary material (e.g., annual 
reports), and any previous FACIT assessments. 

■ Review all relevant material, highlighting 
information that can be used to help formulate 
particular ratings. Make decisions based on the 
totality of the evidence available. 

■ For ratings that are difficult to make, review the 
evidence and the wording associated with each 
possible score. There might not be a perfect 
match available in the possible scoring choices. 
In that case, find the score that best describes 
the item. 
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Conciliation 

The process used to bring FACIT Team members’ 
scores together for a common rating is called 
conciliation. Rather than relying on simple averaging 
of responses from team members, conciliation relies 
on team dialogue to reach a joint decision. 

If team members identify a discrepancy in ratings, 
they should take the following conciliation steps: 

■ Refer to the FACIT scoresheet to help 
understand the issues intended to be rated 
under the specific item. 

■ Share all infonnation used to arrive at the 
individual decisions. 

■ Use all of the data to come to a joint decision 
about the appropriate rating for the program 
on the particular item. 


Special problems 

Certain problems may complicate the conciliation 
process. This section addresses the most common 
difficulties and suggests solutions. It is critical that 
the FACIT Team members rule out all other 
alternatives before assigning a score of “0.” 


Information is inadequate or not available 

Team members should attempt to rate the program 
on every item of the FACIT. In some cases, 
however, they may feel that they do not have 
adequate information. There may be several 
reasons for the lack of information, each of which 
might have its own specific remedy: 

■ During the FACIT evaluation, the team 
neglected to collect certain information or ask 
certain questions of one or more members or 
staff. In this case, the team should attempt, as 
possible and appropriate, to contact the program 
and gather additional information prior to the 
initial scoring of FACIT items. 

■ An item on the FACIT scoresheet is not 
applicable to the program. A FACIT Team 
member may feel that a particular item is not 
applicable. If the team agrees, the N/A (not 
applicable) notation should be written in the 
rating cell and a brief explanation should be 
included. No score is recorded. 

■ The team did not collect any evidence related 
to an item on the FACIT scoresheet because 
the practices being rated are not demonstrated 
within the program. This is distinct from the 
previous situation. If the practices are not in 
evidence but are applicable to the organization, 
they should be scored “I.” In general, the score 
1 should be reserved for practices that are 

not demonstrated. 
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The optimal item rating appears 
to be between two scores 

There may also be times when an assessor believes 
that the tme rating for a particnlar item falls 
somewhere between two available valnes. If the 
FACIT Team member is still in donbt after carefully 
reviewing the options, he or she should assign the 
lower of the two values to the item. 


Data entry 

After coming to an agreement about the ratings, 
team members should total and record them where 
indicated on the FACIT scoresheet and begin data 
entry using the Microsoft Excel Spreadsheet 
provided with this KIT. More information about 
scoring is provided in Appendix B. 


Substantial disagreement between raters 

In the rare cases in which conciliation is 
impossible, each member should write out his 
or her opinion, with references to supporting 
evidence. These documents should be sent to 
a superior or supervisor for a final decision. It is 
more helpful to the program if final ratings are 
determined by the actual evaluators rather than 
by an outside party. Therefore, FACIT Team 
members are urged to conciliate and arrive 
at agreements whenever possible. 
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Conducting an Outcome Evaluation 

This section of the KIT presents the basics of conducting an outcome evaluation. 
Information is provided about the following: 

■ Creating an evaluation plan; 

■ Identifying key outcomes to be measured; 

■ Selecting outcome measures and data collection methods; and 

■ Using the Peer Outcomes Protocol (POP). 

With this information, you can develop an evaluation plan that is focused on 
ongoing quality improvement and tailored to the consumer-operated program. 


Create a plan 


Key principles 


Begin an outcome evaluation by 
developing a long-term plan that 
incorporates peer support and self-help 
principles and addresses critical questions. 


Consumer-operated programs consistently 
apply peer support and self-help 
principles. Therefore, outcome evaluations 
must have these characteristics: 

■ Be consumer-directed; 

■ Empower the program 
and participants; 
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■ Always respect member choice; and 

■ Include meaningful member input. 

Critical questions 

Outcomes, as well as measurement tools and 
methods, should be carefully selected based on 
their value in achieving program goals. As part 
of the selection process, consider the following: 

■ Which outcomes are most closely associated 
with program objectives and values? 

■ Which outcome tools and methods best align 
with program objectives and values? 

■ How can these tools be optimally employed 
to accurately measure the selected outcomes? 

■ How, when, and where should outcomes 
be measured? 

■ Who should be included in the sample group 
to accurately reflect the diversity of the 
membership? 

■ What types of resources (e.g., funding, staff, 
time, and training) are required to conduct 
the outcome evaluation and how will they 
be obtained? 

Appropriately applying the principles and 
answering the questions also requires paying 
thoughtful attention to the issues of confidentiality 
and informed consent. More information about 
these issues is included in the earlier Conducting 
a FAC IT Fidelity Assessment section of 
this booklet. 


Select knowledgeable evaluators 

Outcome data should be collected and evaluated 
by trained personnel. This task may be assigned 
to the program leader or other staff responsible 
for quality improvement. Alternatively, members 
of the program advisory committee or board, as 
well as independent consultants, might be selected 
to take on this responsibility 

The assigned personnel will need to be appropriately 
trained. Many of the commonly used protocols, 
such as POP, include training resources. 


Kinds of outcomes 

Program evaluation involves measuring two types 
of outcomes: 

■ Core outcomes related to members’ growth, and 

■ Program satisfaction outcomes linked to specific 
events, approaches, and activities offered by 
the organization. 

A third set of outcomes also should be considered 
by programs offering specialized services such as 
housing, education or work support, respite, and 
so forth. 
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Core outcomes 

Core outcomes are common to all peer-run 
programs and reflect the program’s general 
operating principles—supporting self-help and 
promoting recovery and resiliency. Core outcomes 
include hope, personal empowerment, sense of 
self or personhood, social connectedness, and 
belief that one’s life has meaning. Standardized 
tools, primarily self-reports, have been developed 
to measure these outcomes in ways that value an 
individual’s unique experiences and accomphshments. 
These tools can be used to generate both short- 
and long-term results. 

Program satisfaction outcomes 

Program satisfaction outcomes look at participants’ 
attitudes toward the organization’s services, 
practices, staff, and environment. They are 
invaluable indicators of how well the program is 
serving its members. Information is collected 
through subjective measures, such as self-reports, 
and objective measures, such as program data 
analyses. These tools indicate that even short-term 
program participation can have positive outcomes. 

Outcomes for specialized programs 

An additional set of outcomes should be measured 
to look at specialized services that target specific 
populations. For example, outcomes related to 
drug and alcohol use should be collected from 
programs providing special services for mental 
health consumers with co-occurring substance use 
disorders. Some of the measures must be repeated 
over an extended period of time before significant 
results are produced. 


Select appropriate outcomes 
for the evaluation 

Identify key core outcomes for the first evaluation. 
Lessons learned from conducting this first study 
can be incorporated into later, more complex 
evaluations. For help identifying the first set 
of outcomes, review the key questions in the 
beginning of this section and think through 
possible ramifications. 

Simpler outcome evaluation methods include 
holding member focus groups or conducting brief, 
one-shot surveys that use established measures 
to determine the impact of services on a limited 
number of participant outcomes. 

Developing a logic model can be extremely 
helpful in selecting appropriate outcomes for more 
developed consumer-operated programs. Basically, 
a logic model analyzes the work of the service 
into four categories or steps: inputs (resources), 
activities, outputs (short-term results), and 
outcomes (longer term impact). 

University of Idaho Extension has produced 
a five-page guide to basic logic models, available at 
http://www.uiweb.uidaho.edu/extension/LogicModel.pdf. 
For a more detailed guide on logic model 
development and use, consult the W. K. Kellogg 
Foundation’s Logic Model Development Guide: 
Using Logic Models to Bring Together Planning, 
Evaluation, and Action. The document is available 
from http://www.wkkf.org/. It includes an 
introduction to model development, sample 
logic models, exercises, and a resource list. 
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Outcome measures 

Work with the evaluator to identify appropriate 
outcome measures and data collection methods 
for the consumer-operated service. Select a 
combination of measures and collection methods 
that work well together and will provide enough 
data to generate significant results. 

It is best to start with simple, straightforward 
outcome measures and data collection methods. 
These may be expanded after the organization s 
leaders and members have become accustomed to 
participating in evaluations and using the findings 
to improve the consumer-operated services. 

Standard protocols 

Consumer-operated programs are highly 
encouraged to adopt a commonly used evaluation 
protocol that has been used to evaluate other 
programs. A protocol is a set of questions to 
be asked or information to be gathered often, and 
a specific approach to be used to collect the data. 

Creating a new, “home-grown” protocol requires 
considerable skill in scale development, testing, 
and analysis to ensure that the protocol is reliable 
and valid. Adopting a standard or commonly 
used evaluation protocol requires far less expertise 
and effort. 


Commonly used protocols may focus on a single 
outcome (e.g., hope) or measure a group of 
outcomes. Many of these protocols have been 
shown to be valid and reliable in evaluating 
consumer-operated services and can be adapted 
for use by different programs. These include 
the following: 

■ Herth Hope Index (Herth, 1991); 

■ Meaning of Life Framework Subscale (Life 
Regard Index, Battista and Almond, 1973); 

■ Subjective Social Inclusion Scale (QOL 
Interview excerpts, Lehman, 1983); 

■ Empowerment/Making Decisions Scale 
(Rogers, Chamberlin, ElHson & Crean, 1997); 
and 

■ Recovery Assessment Scale (Corrigan, Giffort, 
Rashid, Leary, & Okeke, 1999). 

However, even though individual single-outcome 
tools may be valid and reliable, combining them 
psychometrically is more complex than it may 
appear. This approach is feasible for programs with 
sufficient resources to hire a professional evaluator. 

The POP was developed to make evaluation easier 
by providing both single and multiple outcome 
measurement options and by including evaluator 
training. The following section explains the POP 
in more detail. 
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Using the Peer 
Outcomes Protocol 

For many consumer-operated programs, the Peer 
Outcomes Protocol (POP) may be the most useful 
evaluation tool. The POP is consistent with peer 
support philosophy. In addition to being developed 
primarily by consumers, the tool was developed 
to help ensure that consumer values were included 
in outcome measurements. The POP (1) recognizes 
and utilizes proven consumer abilities to conduct 
survey and outcome studies and (2) is designed 
to be administered by consumers. 

The POP has seven independent modules that 
can be used together or separately, so programs 
can measure the outcomes relevant to their goals. 


Each module focuses on one of the following 
topics: demographics, service use, employment, 
community life, quality of life, well-being, and 
program satisfaction. Every outcome module 
includes objective as well as subjective items. 
Self-report items ask specifically about the effects 
of peer support on the particular outcome domains. 

The instrument, a manual describing how to 
administer the POP, a question-by-question 
guide, a set of response cards, and related 
background information can be downloaded 
from the National Research and Training Center 
on Psychiatric Disability (NRTC) Web site: 
http://www.cmhsrp.uic.edu/nrtc/pophome.htm. 

The POP meets key criteria that facilitate the use 
of any outcome evaluation protocol by a consumer- 
operated program. See the text box below. 


The Peer Outcomes Protocol meets criteria for use in consumer-operated services 


Adaptive Utility: Each module may be presented as a separate unit. Programs may collect data 
on a subset of domains. 

Simplicity: The measures included are accessible to, and easily collected by, any community-based 
peer support program. Sophisticated computer technology and advanced expertise in program 
evaluation are unnecessary. 

Flexibility: The protocol offers multiple methods for data collection and management. It provides 
guidelines for matching methodology with program resources, infrastructure, and information needs. 

Face Validity: POP measures are relevant for consumer-operated programs and readily understood 
by nonresearchers. 

Responsiveness and Sensitivity: The POP was developed by and for persons with psychiatric 
disabilities as a tool for collecting outcome information about recovery, personhood, empowerment, 
well-being, employment, and community life. 

Fluman Subjects Protections: The POP places top priority on respecting the rights of people 
from whom data are being collected. The tool promotes adherence to participants' rights to privacy, 
confidentiality, and informed consent. POP principles promote voluntary participation and easy 
access to one's own data and aggregated data reports. 

Collaboration: POP principles emphasize member participation in all aspects of the evaluation; 
this includes planning, data collection and analysis, quality assurance, and the application and 
dissemination of results. 
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Data collection 

Examples of data collection methods are outlined 
below. Select the one (or more) that fits best with 
your outcome measurement tool. Often, more than 
one method is needed to collect enough data in 
a reasonable amount of time. 

In general, data are collected via mail or telephone 
or through in-person interviews with individuals 
or groups. 

Mail surveys are cost-efficient, anonymous, and 
require less time to administer than other data 
collection methods. They can be used to reach 
a broad range of people, including recipients 
who do not have phones or who are unavailable 
for other types of interviews. However, mail surveys 
have low response rates (usually because they are 
time consuming), and they exclude individuals who 
cannot read the language used in the questionnaire. 

Telephone surveys may be an efficient way to reach 
people who do not or cannot respond in writing 
or in person. They may be particularly useful for 
collecting data based on open-ended questions. 
However, telephone surveys are more expensive 
than mail surveys because interviewers must be 
trained to administer the survey. In addition, it can 
be difficult to gain consent during a telephone call 
and the sample will be limited to people who have 
telephone service. 

One-on-one interviews yield the highest response 
rates. They also allow for in-depth probing and 
prompting to obtain data. In addition, the 
informed consent process can easily be used 
to ensure confidentiality. However, personal 
interviews are expensive: interviewers must 
be trained and paid, and scheduling conflicts 
may lead to losses of both time and money. 

Group interviews have some of the advantages 
of one-on-one interviews, but prompting and 
probing for information may be more difficult. 

In many cases, however, these limitations are 
offset by lower administration costs. 


Enter, summarize, analyze, 
and interpret data 

Once data are collected, they must be coded 
and entered into the data analysis system using a 
standardized process. Initially, data may be entered 
and summarized using a simple report form and 
hand-tallying system. However, a computerized 
system that includes commonly used software such 
as SPSS or Access will greatly simplify collecting and 
analyzing data and preparing reports as the program 
grows. Over time, it may be most cost-effective to 
collect data in an electronic format and outsource 
more complex data analysis and report preparation 
to private firms or local academic institutions. 
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Reporting and Using Results 

As the program develops its quality improvement efforts, fidelity assessments 
and outcome evaluations become invaluable assets for developing a program 
truly responsive to consumer needs and interests. As members' familiarity with 
assessment and evaluation principles and activities grows, you may wonder 
how the program ever operated without these processes in place! 

This section focuses on reporting and using FACIT results for quality improvement. 
Additional information about using Peer Outcomes Protocol as part of the evaluation 
process is available at the National Research and Training Center on Psychiatric 
Disability Web site, http://www.cmhsrp.uic.edu/nrtc/pophome.htm 


Baselines and benchmarks 

The first fidelity assessment establishes 
the program baseline and is a starting 
point for efforts to become an evidence- 
based peer practice. The baseline scores 
can help you identify or confirm the 
existence of program strengths and 
weaknesses, especially when compared 
with the national FACIT benchmarks. 


The national benchmarks are the 
aggregated FACIT scores from the 
eight consumer-operated programs that 
participated in the Consumer-Operated 
Services Program (COSP) Multisite 
Research Initiative. Since these programs 
were geographically diverse, represented 
multiple program models, and met the 
federal criteria for a consumer-operated 
program, their aggregated FACIT scores 
are a useful standard for gauging 
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individual program performance. More 
information about comparing program and 
benchmark scores is included in Appendix C. 

Over time, comparisons of fidelity assessment 
results for the program will help you track the 
program’s progress toward meeting its goals. 
Comparisons with the national benchmarks also 
will provide insight about how well the program 
is doing. 

This KIT will enable you to make comparisons 
over time using traditional bar graphs and 
longitudinal plots. The graphs and plots are 
powerful feedback tools because they help 
program leadership develop an accurate long- 
range perspective on participation and outcomes. 

You can use a longitudinal plot to track results 
for a specific outcome, a set of related outcomes 
(domain), or an entire program. A single plot 
can also contain longitudinal data for multiple 
programs or groups of members, enabling the 
leadership to make meaningful comparisons. 


Share your results 

The single factor that will most likely determine 
the success of quality improvement efforts is the 
ability to give useful and timely feedback to 
program members and staff. To influence program 
and individual practices and to help various 
audiences appreciate the value of conducting 
fidelity assessments, the data must be (1) translated 
into understandable and meaningful information 
and (2) tailored for each specific audience. 

Prepare reports explaining the assessment results 
and share them with the program participants at 
member and staff meetings. Also report the results 
to your advisory board or board of directors and 
seek their feedback. 

By sharing the fidehty reports, you can help create 
a learning organization characterized by adaptive 


responses to information that aims to improve 
program operations and member outcomes. 

In addition, you can gain greater credibility 
with, and support from, other stakeholders 
within the continuum of care by sharing evaluation 
information, including FACIT reports, with them. 


Writing FACIT reports: 

An overview 

FACIT reports presenting different types of 
information may be needed for different audiences. 
For example, the board of directors may want to 
see everything, but an annual report to the funder 
may focus on only key successes and challenges. 

The following overview is meant to guide 
the writing of a general report. However, this 
information can be adapted for drafting other 
types of FACIT reports. 

Purpose of the report 

The team or individual responsible for quality 
improvement should write a report using the 
FACIT scores to help identify the program’s 
strengths and weaknesses and offer 
recommendations for program improvement. 

The report should be informative, factual, and 
constructive. It should be based in part on input 
from members and other evaluation participants. 
The report should be simple and reader friendly 
and provide infonnation about both the overall 
program and specific topics. 

The Excel program provided with this KIT 
wifi display data in both numerical and graphed 
formats. The graphic data displays are powerful 
tools. Coupled with explanatory text, they can be 
used to create effective formal documents (such 
as reports and slide presentations) that explain 
the program’s strengths, weaknesses, and goals 
to members, advisors, and other key audiences. 
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Use tables and graphs to explain 
your data 


Visual representations can make it easier for 
readers to understand key points in a report. 

The graphs and tables illustrate key FACIT results 
for consumer-operated programs. For example, 
Figure 1 is an Excel-generated bar graph showing 
first year or baseline results for individual peer 
practices within the structure domain. 


Figure 1 also compares first year results to the 
national FACIT benchmarks. Notice how easy it is 
to see how the two compare. Strengths and 
weaknesses as compared with the national 
benchmarks are apparent, making it easier 
to identify priorities for program development 
and quality improvement planning. 


Figure 1. Sample Fidelity Baseline Assessment/ 
Structure Domain 



Year 1 
Benchmark 
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As additional assessments are eondncted over time, 
bar graphs can be generated that display changes 
in the program’s FACIT resnlts. Two examples are 
inclnded below. 

Fignre 2 displays the resnlts of one program’s three 
annnal assessments of program stmctnre highlighting 
the individnal peer practices within the stmctnre 
domain. Fignre 3 shows the scores in all program 
domains for three years of annnal assessments. 


Both figures also provide visual comparisons with 
the national benchmarks. 

This chart makes it easy to see changes on the 
same item from year to year and also to compare 
them with the national benchmark. Is the program 
moving closer toward fidelity? Where are the most 
gains? What areas should be prioritized for 
further development? 


Figure 2. Fidelity Assessment Over Time/ 
Structure Domain 


ns 


Item 



Year 1 
Year 2 
Year 3 
Benchmark 


Reporting and Using Results 


30 


Evaluating Your Program 
















































Figure 3. Fidelity of All Domains Over Time/ 

Comparison of Domain Scores to National Benchmark 
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The bar graphs are easily generated. They may be 
supplemented with other outcome displays; 
however, most of these are dependent on 
sophisticated analyses and will probably require 
support and consultation to “crunch the numbers” 
and produce the reports. One exception is the 
longitudinal plot. 

Figure 4 is a longitudinal plot comparing levels of 
well-being for two groups of consumers, those who 
did and those who did not attend one of the eight 


programs that participated in the COST Multisite 
Research Initiative. One group visited the program 
at least once and the other never visited during the 
12-month study. The longitudinal plot shows that 
participation in a consumer-operated service 
program (at least one visit) is associated with 
greater well-being outcomes. 


Figure 4. "Well-Being" Means for Engagement in a Consumer-Operated Service 



Time Since Baseline (in months) 
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Additional resources 


This brief overview is meant to give you some ideas 
and practical advice for proceeding with fidelity 
assessments and outcome evaluations. 

Additional resources are available at the National 
Research and Training Center on Psychiatric 
Disability Web site: http://www.cmhsrp.uic.edu/nrtc/ 
and in the Evidence booklet of this KIT. 
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EVIDENCE-BASED 
PRACTICES 

Kfr 

Knowledge Informing Transformation 
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For technical assistance on FACIT materials contact: 

Jean Campbell, Ph.D. 

Director, Program in Consumer Studies and Training 
Missouri Institute of Mental Health 
5400 Arsenal Street, St. Louis, MO 63139 
(314) 877-6457 
jean.campbell@mimh. edu 
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Fidelity Assessment Common 
Ingredients Tool 

Facit Protocol 
June 2007 

Introduction 

The FACIT review includes two important elements: 

1. Observing the physical environment and existing 
social relationships; and 

2. Using structured questions to interview program 
leaders, staff, and members. 

The evaluators use information from both the 
observations and interviews to determine the 
program scores that they then record on the 
FACIT scoresheet. Some of the elements to 
be observed are discussed below. The structured 
questions to be used in interviews begin on 
page 40. 

FACIT Team observations 

As part of the collection of evidence about the 
presence of the common ingredients, the FACIT 
Team will tour the program facility and observe 
the program activities and environment. FACIT 
Team members are particularly encouraged 
to observe and take notes regarding common 
ingredients not readily available through 
the interview or focus group. 


Some things to observe include the following: 

Safety 

■ Written rules of behavior or policies of expulsion 
are non-coercive. 

Physical Environment 

■ The furniture looks comfortable. 

■ Peers appear relaxed and at ease 
with each other. 

Social Environment 

■ Staff is out on the floor interacting with 
members. 

■ One might not be able to distinguish staff 
from members by clothing or attitude. 

■ Everyone is on a first name basis. 


Sense of Community 

■ People know each other by name. 

■ Members come together to make decisions 
about the program. 

■ There are photos of members at retreats and 
other social events such as camping trips 
and celebrations. 

■ A newsletter published by members covers the 
organization s activities and community events, 
and may feature art, poetry, and prose. 
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Acceptance and Respect for Diversity 

■ In member interactions there appears to be 
an acceptance of differences and a willingness 
to learn about differences. 

■ There is minimal usage of clinical labels in 
conversation and in written materials about 
the consumer-operated service. Stigmatizing 
language is eschewed. 


Spiritual Growth 

■ People are able to talk about spiritual growth, 
be accepted, and not considered to have 
religious delusions. 


Peer Support 

■ Newsletters, mission statements, and other 
documents convey peer support principles. 

■ There are displays of written stories, poetry, 

or artwork created by members as part of telling 
their stories. There are a variety of avenues to 
tell stories. There is evidence that consumers 
have control over when and for whom they tell 
their stories. 

■ Members are encouraged to work to help other 
peers and to move the consumer 
movement forward. 

■ Materials about the history of the consumer 
movement are available, and members have 
access to documents, newsletters, position 
papers, and written testimony from other 
consumers and consumer organizations. 


Peer Mentoring and Teaching 

■ There is evidence of formal matching 
of members by staff. 

■ An established leadership exists within peer 
support groups and classes. 


Recovery 

■ Newsletters, mission statements, and other 
program documents convey a recovery 
philosophy. 

■ Self management problem-solving strategies 
are in evidence. 

■ Fonnal educational programs are available 
for problem solving, such as anger 
management classes. 

■ There is evidence of informal exchange of 
personal experience to enhance individual 
problem-solving abilities. 


Formal Skills Practice 

■ Formal or informal teaching and practice of 
daily living skills, vocational skills, job readiness, 
communication skills, goal setting, and 
assertiveness skills occurs on a regular basis. 
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Interviews and focus groups: 
Suggested introductory script 

Introduction 

Hello, our names are_ 

and we are members of the FACIT Team. 

FACIT stands for the Fidelity Assessment 
Common Ingredients Tool. 

Our team is gathering information about the 
organization to find out how well the program 
is delivering peer support services. 

The results of our work will be shared with 
members and staff of your consumer-operated 
program so you can develop a plan to improve 
some things you do, add activities and new 
ways to offer peer support, and celebrate 
your successes. 

Like many of you, "I have also received mental 
health services" (or "I attend a consumer- 
operated program"). 

Explain purpose 

As program members (or staff) you have been 
asked to participate in this focus group to talk 
about the things the program is doing or needs 
to be doing to create a place where consumers 
can grow, change, and reclaim productive lives 
in their communities. 

Many people use the term recovery to describe 
this process of growing, changing, and claiming 
our place back in our communities. 

We will tape record our discussion, but the tape 
will not be shared with anyone but the FACIT 
Team members. 


We will use the tape to check what was said if we 
miss something in our notes or disagree with each 
other about what was said. The tape will be stored 
in a locked cabinet and destroyed after we report 
the results to your program. 

Clarify and answer questions 

ASK: Do you have any questions? 

[Note: Take time to make sure all questions are 
fully answered to the participants satisfaction.] 

Let's take a few minutes and have each of you 
introduce yourselves. Please just use your 
first names. 

[Note: Use this time to create a comfort zone 
for participation.] 

Explain guidelines for discussion 

Before we begin the focus group, there are 
a few guidelines we’d like you to follow: 

■ Say whatever you think is true for you. 

There are no “right” or “wrong” answers. 

■ Make as many comments as you want because 
we are trying to generate as many responses 
as possible. 

■ Respect the person who is speaking, do not 
interrupt him or her, and do not criticize 
anyone else’s ideas. We can ask each other 
for clarification though, if we need to. 

■ Give everyone a chance to share his or her 
own ideas by speaking briefly and sticking 
to the point. 
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■ Speak from your own experiences. 

■ Personal information that is shared here 
stays here. 

[Note: Often people will say they want what is said 
in the room to stay in the room. This gives people 
ownership over the rules. Posting the rules is good 
because, if someone is monopolizing or criticizing 
another person s input, then the facilitator can 
point to the rules and gently remind him or 
her—“this is what we all agreed to.”] 

Does anyone want to add any other rules 
so you all can feel comfortable participating? 

Now we can begin the focus group discussion. 


Semi-structured interview/focus 
group questions 

The following sections provide general and FAC IT- 
specific questions for the program; 

■ Director; 

■ Supervisory staff; 

■ Nonsupervisory staff; and 

■ Member focus groups. 

Questions for the director 


General questions: 

■ What expectations should a consumer have 
about being a member/participant here? 

■ In what way are members expected to be 
accountable for their actions here? 

■ How is the pace at which a member wishes 
to participate or be involved in the 
program respected? 

■ How is recovery an important component here, 
and how is it defined? 

■ How is spirituality viewed, and how is it 
expressed in the program? 

■ In what ways do members have a chance to 
“tell their stories”? 

■ Do participants receive information about the 
consumer movement here? 

■ How is crisis prevention handled here? 

■ What type of emphasis is placed on participants 
being able to identify their own needs? 
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■ How do participants learn to deal effectively 
with entitlement agencies (such as the Social 
Security Administration or HUD)? 

■ How would you describe your relationship 
with traditional mental health providers in 
your community? 

■ If you could change anything you wanted about 
the program, what would you change first? 


FACIT questions: 

Board Participation 

Do you have a board or decisionmaking group? 

If so— 

■ Is the board (or decisionmaking group) elected, 
and by whom? 

■ What percent of the board (or decisionmaking 
group) identify themselves as consumers? 

■ Does the board (or decisionmaking group) make 
binding policy? 

■ How often does the board (or decisionmaking 
group) meet? 

Consumer Staff 

■ What percentage of the staff identify 
as consumers? 

■ What staff/leadership positions 
do consumers hold? 

Hiring Decisions 

■ Is there a written policy regarding who 
can be hired as staff in the program? 

■ Who is involved in firing decisions? 


Budget Control 

■ Who develops the budget? Are 
consumers involved? 

■ Who signs checks? Is this a consumer? 

■ Who decides the salaries of the staff? 

Are consumers involved in these decisions? 

■ Who is authorized to sign contracts on behalf 
of the agency? Is this a consumer? 

Planning Input 

■ Are there organizational or strategic planning 
meetings? If yes, how is member input solicited? 

■ Are there meetings in which members can 
state their desired changes for the program? 

Are requested changes addressed and 
accommodated as soon as possible? 

■ Are there changes that staff or members would 
like to make but that your funding sources or 
laws/regulations preclude you from making? 

Satisfaction/Grievance 

■ Does your program have a grievance policy? 

■ Does your program conduct 
satisfaction surveys? 

■ Are there forums at which members can raise 
their concerns about the program and 

offer suggestions? 

Traditional Mental Health Services 

■ To what extent do you exchange information 
with traditional service providers? Provide 
referrals to them? Receive referrals from them? 
Exchange resources with them? 
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Lack of Coerciveness 

■ Are there any program requirements for 
member participation in the program or peer 
support groups? 

Recovery 

■ Are there members who no longer participate 
in the program because they got the support 
they needed from the program (such as became 
employed or managed their symptoms)? 

■ What is the program doing to promote recovery 
or to show that it embraces recovery? 

Crisis Prevention 

■ What are some of the ways that the program 
prevents crises or helps members who are 
in crisis? 

Questions for supervisory staff 

Note: If there is no supervisory staff in this 

program, these questions may be answered 

by the program leadership. 


General questions: 

■ How long has each of you worked here? 

■ What expectations should consumers/ 
participants have about receiving services here? 

■ What is the most important thing you can 
do for a member/participant? 

■ In what ways are members accountable 
for their actions here? 

■ How is the pace at which the member wishes 
to be involved respected here? 

■ What is the importance of recovery here 
and how is it defined? 


■ How is spirituality viewed here, and how 
is it supported in the program? 

■ When do members have a chance 
to “tell their stories”? 

■ Do members receive information about 
the consumer movement here? 

■ How is crisis prevention practiced here? 

■ What type of emphasis is placed on participants 
being able to identify their own needs? Is staff 
responsive to those needs? 

■ How do participants learn to deal effectively 
with agencies responsible for entitlement 
programs (such as the Social Security 
Administration or HUD)? 

■ How would you describe your relationship 
with the traditional mental health 
service community? 

■ If you could change anything you wanted about 
the program, what would you change first? 


FACIT questions: 

SatisfactionIGrievance 

■ Are there forums at which members can raise 
their concerns about the program and make 
suggestions or recommend changes? 

■ How are personal and program concerns 
of members addressed by the program? 

Traditional Mental Health Services 

■ To what extent does the program exchange 
information with traditional service providers? 
Provide referrals to them? Receive referrals 
from them? Exchange resources with them? 

■ To what extent does staff participate in local 
service network meetings? 

■ Are members forced or coerced to follow 
through with referrals? 
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Local Proximity 


Accessibility 


■ Is the program located within walking distance 
of the center of town or where there is a major 
concentration of consumers? 

Access 

■ How close is the program to the nearest 
bus line? 

■ Does the program provide or arrange 
for transportation to/from the program? 

■ Does the program supply bus tickets 
or taxi vouchers? 

Hours 

■ What are the program’s hours of service? 

■ Which days is the program open? 

■ How are decisions on the hours/days 
of operation made? 

■ Are there arrangements that members can make 
to receive support after hours? 

■ Does the program remain open after other 
programs close? 

■ Is the program open on evenings, weekends, 
and/or holidays? 

Cost 

■ What fees are there for program 
membership/attendance ? 

■ What fees are there for participating 
in any program activities? 


■ Is the program accessible for members with 
physical disabilities? Does it have wheelchair 
ramps and necessary accommodations in the 
bathrooms available? 

■ Are program materials prepared in various ways 
to accommodate members with physical 
disabilities such as offering Braille or large print 
versions of written materials? 

■ Does the program have a TTY system available 
for individuals who need it? Does the program 
have a contract to use relay services? 

■ To what extent do persons with disabilities other 
than psychiatric disabilities actually attend the 
program and participate in activities? 

Lack of Coerciveness 

■ What requirements are there for participation 
in program activities? 

Sense of Community 

■ How does program staff link individual 
members together for mutual support 
or activities? 

Peer Principle 

■ Do staff members disclose their psychiatric 
disabilities to other members? 

■ Have staff shared their recovery stories 
with other members? 

Helper's Principle 

■ Describe a recent experience in which you 
helped someone at the program. 

■ Was this experience helpful in any way 

for you, as well as being helpful to the person 
you helped? 
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Empowerment 

■ How has being involved as a staff member at 
the program helped you make positive changes 
in your life? 

■ What was it that helped you make 
these changes? 

■ How does program participation help you feel 
that you have more control over your life? 

■ How does participation help you feel that 
change in the mental health system is possible? 

Group Empowerment 

■ Do you feel pride in being a program member? 

■ Do you feel that you can contribute/make 
an impact on the program? 

Member Choice 

■ What types of programs or activities does 
the program offer? 

Recovery 

■ What is the program doing to promote recovery 
or to show that it embraces recovery? 

Spiritual Growth 

■ Are there opportunities to share 
spiritual beliefs? 

■ What happens if someone shares their spiritual 
beliefs with other members? 

Exploration of Meaning and Purpose 

■ What activities, such as writing, artwork, or 
poetry, are there to help or encourage people 
to express meaning and purpose in life? 


Formal (Scheduled) Peer Support Groups and 

Individual Peer Support Relationships 

■ How often do formal peer support groups meet? 

■ How many members participate in formal 
individual or group peer support? 

Informal (Unscheduled) Peer Support Groups and 

Individual Peer Support Group Relationships 

■ How often do informal peer support 
groups meet? 

■ How many members participate in informal 
individual or group peer support? 

Crisis Prevention 

■ What are some of the ways that the program 
helps people who are in crisis? 

Peer Mentoring and Teaching 

■ How has someone at the program been 
a mentor to you? 

Job Readiness Activities 

■ Please provide examples of program efforts to 
improve: (1) communication skills or confidence 
of participants, (2) assistance in preparing 
resumes, (3) practice in employment interviews, 
and (4) support in setting up 

employment interviews. 

Formal Self-Advocacy Activities 

■ Does participation in this program help 
members to obtain services from other 
community agencies? 
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Questions for nonsupervisory staff 


FACIT questions: 


General questions: 

■ How long has each of you worked here? 

■ What expectations should consumers have 
about receiving services here? 

■ What is the most important thing you can 
do for a consumer? 

■ In what way are members expected to 
be accountable for their actions here? 

■ How is the pace at which the member wishes 
to participate or be involved respected here? 

■ What is the importance of recovery here, 
and how is it defined? 

■ How does this program support members 
in using spirituality as part of their recovery? 

■ How do members have a chance to “tell their 
stories” of emotional distress and recovery here? 

■ Do members receive information about 
the consumer movement here? 

■ How is crisis prevention handled here? 

■ How does the program help members 
to identify their own needs and make 
sure that those needs are taken care of? 

How do members have a chance to learn 

to deal effectively with entitlement agencies 
(for example, HUD or Social Security)? 

■ If you could change anything you wanted about 
this program, what would you change first? 


Hiring Decisions 

■ Is there a written policy regarding who can 
be hired as program staff? 

Budget Control 

■ Who decides the salaries of the staff? 

Are any consumers involved in these decisions? 

Volunteer Opportunities 

■ Is there an opportunity to become staff after 
participating in the program or volunteering 
in it? 

Planning Input 

■ Are there membership meetings in which 
members can state their desired changes? 

■ If members request changes, are these requests 
reviewed and adopted as appropriate? 

Satisfaction/Grievance 

■ How are concerns of members addressed? 

Traditional Mental Health Services 

■ To what extent do you exchange information 
with traditional service providers? Provide 
referrals to them? Receive referrals from them? 
Exchange resources with them? 

■ To what extent does staff participate in local 
service network meetings? 

■ Are members ever forced or coerced to follow 
through with referrals? 
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Accessibility 

■ To what extent do persons with disabilities 
(other than psychiatric disabilities) actually 
participate in activities at the program? 

Lack of Coerciveness 

■ Are there program rules for members? 

How were they developed? How are they 
shared with members? 

Sense of Community 

■ As staff, how do you link or bring together 
individual program members? 

Peer Principle 

■ Does staff disclose to members that they have 
psychiatric disabilities? 

■ Does staff share their recovery stories 
with members? 

Helper's Principle 

■ Describe a recent experience of helping 
a member. 

■ Was this experience meaningful for you? Why? 

Empowerment 

■ How has being involved in the program helped 
you make positive changes in your life? 

■ How were you able to make these changes? 

■ How has participation in the program enabled 
you to have more control over your life? 

Group Empowerment 

■ Do you feel pride in being a program member? 

■ Do you feel that you are making an impact 
on the program? 


Member Choice 

■ What types of activities are offered at 
the program? 

■ Do members have the option to select from 
a variety of activities? 

Recovery 

■ Are there members who have moved on 
with their lives because they got the help 
they needed from the program? 

■ Are there former members who left the 
program because they became employed? 

■ How do you define recovery? 

■ What does the program do to promote recovery 
or to show that it embraces recovery? 

Spiritual Growth 

■ Are there opportunities for members to share 
their spiritual beliefs? 

■ Do members feel comfortable discussing their 
spiritual beliefs? 

Exploration in Meaning and Purpose 

■ Are there activities that help or encourage 
members to explore meaning and purpose in 
their lives, such as through writing, artwork, 
or poetry? 

Formal (Scheduled) Peer Support Groups and 

Formal Individual Peer Support Relationships 

■ How often do formal peer support groups meet? 

■ How many members participate in formal 
individual or group peer support? 
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Informal (Unscheduled) Peer Support Groups and 

Informal Individual Relationships 

■ How often do these groups meet? 

■ How many consumers participate in informal 
individual or group peer support? 

Crisis Prevention 

■ What are some of the ways that the program 
has helped members who were in crisis? 

■ Has participation in the program helped 
members avoid crises? 

Peer Mentoring and Teaching 

■ Is there anyone at the program who has been 
a mentor to you? Are you a mentor to 
other members? 

■ Please provide examples of program efforts to 
improve: (1) communication skills or confidence 
of members, (2) assistance in preparing resumes, 
(3) practice in employment interviews, and (4) 
support in setting up employment interviews. 

Formal Self-Advocacy Activities 

■ Does participation in this program help members 
obtain services from other community agencies? 


Questions for consumer focus groups 

General questions: 

■ How would you describe this consumer- 
operated program to a friend, also a mental 
health consumer, who is considering 
coming here? 

■ What expectations should consumers have about 
receiving services here? 

■ How are you and other members expected 
to be accountable for your actions here? 

■ How is the pace at which you wish to participate 
or be involved in program activities 
respected here? 


■ How is recovery an important component here 
and how is it defined? 

■ Is spirituality an important component of the 
program, and how is it supported at the program? 

■ How do you have a chance to “tell your stories” 
of distress and recovery here? 

■ Do you receive information about the consumer 
movement here? 

■ How is crisis prevention handled here? 

■ How does staff encourage members to identify 
their own needs and desires and how does the 
program make sure that desired changes are 
positively addressed? 

■ How do you learn to deal effectively with 
agencies providing entitlements (such as 
HUD or Social Security) at the program? 

■ What is helpful about coming to the program? 
What is difficult about coming here? 

■ Tell me about one thing you would like 
to change at the program. 


FACIT questions: 

Member Input 

■ Are there membership meetings in which 
members can state their desired changes 
at the program? 

■ If changes are desired, how does the program 
respond to these requests? 

■ Have there been times that you have been 
pressured to participate in any program 
activities that you didn’t want to attend? 

Program Rules 

■ Do you feel safe at the program? 

■ Are you aware of any program rules that apply 
to member participation? 
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Sense of Community 


Member Choice 


■ How does staff try to link or bring together 
individual program members? 

■ When you are not at the program, do you 
participate with other members in any activities 
that are not part of the program? 

■ Do you feel connected or share a bond 
with other people at the program? 

Peer Principle 

■ Does staff disclose their psychiatric disability 
to program members? 

■ Does staff share their stories of distress 
and recovery with members? 

■ Do you share your experiences of having 
psychiatric disabilities with each other? 

Helper's Principle 

■ Can you describe an experience of helping 
someone out recently while at the program? 

■ How was this experience helpful to you? 

■ Do you ever feel pressured to follow 
the advice of your program peers? 

Empowerment 

■ Has being involved in the program helped 
you make any positive changes in your life? 

How were you able to make these changes? 

■ How does participation enable you to have more 
control over your life? 

■ Does participation in the program make you 
feel that change in the mental health system 
is possible? 


■ What types of activities are offered at the 
program? Are these activities that you usually 
chose to participate in? 

■ Do you have a choice of various activities 
when attending the program? 

Recovery 

■ Do you know of any former members who 
have left the program because they 
became employed? 

■ As a program member, do you have the choice 
when to leave the program or how long to remain? 

■ Describe what the program is doing to promote 
recovery or to show that it embraces recovery. 

Spiritual Growth 

■ Are there opportunities to share your spiritual 
beliefs with other program members? 

■ Do you feel comfortable discussing your 
spiritual beliefs with others at the program? 

Exploration in Meaning and Purpose 

■ What activities help or encourage you to explore 
meaning and purpose in your life through 
writing, artwork, or poetry? 

Crisis Prevention 

■ What are some of the ways that the program 
has helped members that are in crisis? 

■ How has participation in the program helped 
you stay out of the hospital and use alternative 
methods to help you through a crisis? 


Group Empowerment 

■ Do you feel pride in being a program member? 

■ Do you feel that you can contribute to or make 
an impact on the program? 
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■ How has participation in the program helped 
you avoid crisis? 


Peer Mentoring and Teaching 

■ Is there anyone at the program that has been 
a mentor to you? In what way? 


Job Readiness Activities 

■ Can you provide any examples of efforts at 
the program to: (1) help you improve your 
communication skills or confidence, (2) assist 
you in preparing resumes, (3) help you practice 
employment interviews, or (4) support you 
in setting up employment interviews? 


Formal Self-Advocacy Activities 

■ Are you more assertive when receiving services 
from traditional mental health agencies because 
of your participation in the consumer- 
operated program? 

■ Do you feel that you are effective in obtaining 
services from other community agencies due 
to participation in the program? 
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EVIDENCE-BASED 
PRACTICES 

Kfr 

Knowledge Informing Transformation 



Evaluating Your Program 


Appendix B: FACIT Scoresheet 

For technical assistance on FACIT materials contact: 

Jean Campbell, Ph.D. 

Director, Program in Consumer Studies and Training 
Missouri Institute of Mental Health 
5400 Arsenal Street, St. Louis, MO 63139 
(314) 877-6457 
jean.campbell@mimh. edu 
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Scoresheet 


Program Name. 
Site Visitor _ 


Site Visit Date. 


Conciliated Score. 


Score 


Ingredient 

Definition 

Anchored Scale 

Assigned Score 

1.STRUCTURE 

■■ 



1.1. Consumer Operated I 

1.1.1. 

Board 

Participation 

Consumers constitute 
the majority (at 
least 51 %) of the 
board or group that 
decides policies 
and procedures. 

■ No member of the board is self-identified as a consumer. 

■ Up to 50% of the board members self-identify as consumers. 

■ 51 % of the board members self-identify as consumers but less than 51% 
of the officers self-identify as consumers. 

■ 51 % or more of the board self-identify as consumers and more 
than 51% of the officers self-identify as consumers. 




■ 90-100% of the board members self-identify as consumers and all of the 
officers self-identify as consumers. 


1.1.2. 

Consumer Staff 

With limited 
exceptions, staff 
consists of consumers 
who are hired by and 
operate the consumer- 
operated service. 

■ No staff member self-identifies as a consumer. 

■ Up to half of staff self-identifies as consumers. 

■ 51 % or more of staff self-identifies as consumers, but less than 51 % of 
administrators self-identify as consumers. 

■ 51 % or more of the staff self-identifies as consumers and more than 

51 % of administrators self-identify as consumers. 

■ 80-100% of staff self-identifies as consumers and all administrators self- 
identify as consumers. 


1.1.3. 

Hiring Decisions 


■ Consumers are not involved in any hiring decisions. 

■ Consumers have some involvement in hiring decisions. 

— 



■ Consumers are responsible for making most of the hiring decisions (50% 
or more). 




■ Consumers are responsible for making all hiring decisions. 


1.1.4. 

Budget Control 

Consumers 
have control of 
the consumer- 
operated service's 
operating budget. 

■ Consumers are not involved in the development or control of the budget. 

■ Consumers have some involvement in the development and control of 
the budget. 

■ Consumers are responsible for the development and control of most of 
the budget. 




■ Consumers are responsible for the development and control of the 
entire budget. 


1.1.5. 

Volunteer 

Opportunities 

Volunteer 
opportunities for 
consumer-operated 
program members 
may include board 
and leadership 
positions, unpaid 
jobs, and paid 
staff positions. 

■ No consumers serve as volunteers at the consumer-operated program. 

■ Less than 25% of the volunteers are self-identified as consumers. 

■ Between 26% and 50% of the volunteers are self-identified 

as consumers. 

■ Between 51 % and 75% of the volunteers are self-identified 

as consumers. 

■ Between 76% and 100% of the volunteers are self-identified 

as consumers. 
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Ingredient 


Definition 


Anchored Scale 


Assigned Score 


1. STRUCTURE 

1.2. Participant Responsive 


1.2.1. The consumer¬ 
planning Input P'° 9 rann 

is responsive to the 
needs and preferences 
of its members. 

■ There are no realistic opportunities for member input. - 

■ There are some opportunities for member input, but the consumer- 
operated program does not display a commitment to implementing 
recommended changes. 

■ There are some opportunities for member input but the program 
displays minimal commitment to implementing recommended changes. 

■ There are many opportunities for member input and the program 
displays a commitment to implementing recommended changes. 

■ There are multiple avenues for member input and the consumer- 
operated program displays a significant commitment to implementing 
recommended changes. 

1.2.2. Members have formal 

Member 

Dissatisfaction/ dissatisfaction with 

Grievance “nsumer- 

Response operated program 

and to have 
grievances addressed. 

■ There are no formal or informal opportunities to express grievances - 

or dissatisfaction with the consumer-operated program. 

■ There are some formal or informal opportunities for members to express 
grievances or dissatisfaction with the consumer-operated program, 

but the organization does not display a commitment to implementing 
desired changes. 

■ There are some formal or informal opportunities for members to express 
grievances or dissatisfaction with the consumer-operated program, 

but the organization displays minimal commitment to implementing 
desired changes. 

■ There are many formal or informal opportunities for members to 
express grievances or dissatisfaction with the consumer-operated 
program and the organization displays a commitment to implementing 
desired changes. 

■ The consumer-operated service program has a formal, written policy 
for addressing grievances; regularly assesses member satisfaction and 
holds membership meetings; and displays a significant commitment 
to implementing desired changes. 
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Ingredient 


Definition 


Anchored Scale 


Assigned Score 


1 STRUCTURE 

1.3. Linkage to Other Supports 


1.3.1. 

Linkage with 

Traditional 

Mental Health 

Services 

■ The consumer-operated service has no reported linkage to traditional - 

mental health services. 

■ There is a report of minimal linkage with traditional mental 
health services. 

■ There is a report of moderate linkage with traditional mental 
health services. 

■ The consumer-operated service has reported intense linkage 
with traditional mental health services, but this involvement is 
not reciprocated. 

■ The consumer-operated service has reported intense linkage with 
traditional mental health services and this involvement is reciprocated. 

1.3.2. 

Linkage with 

Other Consumer- 
Operated 

Services 

■ The consumer-operated service has no reported linkage to other - 

consumer-operated services. 

■ There is a report of minimal linkage to other consumer-operated services. 

■ The consumer-operated service has reported moderate linkage to other 
consumer-operated services. 

■ The consumer-operated service has reported intense linkage to other 
consumer-operated services, but this involvement is not reciprocated. 

■ The consumer-operated service has reported intense linkage to other 
consumer-operated services and this involvement is reciprocated. 

1.3.3. 

Linkage with 

Other Service 

Agencies 

■ The consumer-operated service has reported no linkage with other - 

service agencies. 

■ The consumer-operated service has reported minimal linkage with other 
service agencies. 

■ The consumer-operated service has reported moderate linkage with other 
service agencies. 

■ The consumer-operated service has reported intense linkage with other 
service agencies, but this involvement is not reciprocated. 

■ The consumer-operated service has reported intense linkage with other 
service agencies and this involvement is reciprocated. 


TOTAL 
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Ingredient 


Definition 


Anchored Scale 


Assigned Score 


2. ENVIRONMENT 
2.1. Accessibility 

2.1.5. 

Accessibility 


Efforts are made 
to ensure that 
consumers with 
physical and 
sensory as well as 
psychiatric disabilities 
can participate in 
consumer-operated 
service programming. 


The consumer-operated service pays no attention to accommodations 
of persons with physical and sensory disabilities. A gross lack of 
accessibility is readily apparent to observers. 

The service makes some provisions for persons with physical/sensory 
disabilities, but gaps in accessibility may create barriers for some 
potential participants. 

The service is generally accessible but improvements can be imagined 
(e.g., the consumer-operated service has accessible entrance and toilets 
but lacks TTD). 

The consumer-operated service is fully accessible to persons with a wide 
range of disabilities, and the program is committed to accommodating 
individual differences. 


2.2. Safety 

2 . 2 . 1 . 

Lack of 
Coerciveness 


The consumer- 
operated service 
provides a non- 
coercive, safe 
milieu. Fears due 
to past trauma are 
appreciated and 
assuaged, including 
trauma induced by 
the mental health 
system. 

There are no threats 
of commitment, 
clinical diagnoses, 
or unwanted 
treatments forced 
on members except 
in cases where suicide 
or physical danger 
to other participants 
is imminent. 


Members are required to be in formal treatment to participate 
in the consumer-operated service. 

The consumer-operated service strongly encourages, but does 
not require, members to be in formal treatment to participate. 

The consumer-operated service strongly encourages members 
to participate in peer support programs. 

The consumer-operated service encourages members to participate 
in peer support programs. 

The consumer-operated service encourages members to choose whether 
or not to participate. 


2 . 2 . 2 . 

Program Rules 


Norms/rules to 
protect the physical 
safety of participants 
are developed by and 
for consumers — either 
by the members or 
by consumer staff— 
and they are agreed 
to by all participants. 


Inadequate controls/safeguards. Members frequently feel unsafe 
or are victims of crimes. 

Inadequate controls/safeguards. Members sometimes feel unsafe 
or are victims of crimes. 

Adequate controls/safeguards. Members feel safe from physical harm/ 
crime. Rules not developed by members. 

Adequate controls/safeguards. Members feel safe from physical harm/ 
crime. Rules developed by members. However, no procedures in place 
to address violations of rules. 

Adequate controls/safeguards. Members feel safe from physical harm/ 
crime. Rules developed by members and procedures in place for when 
rules are violated. 
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Ingredient 


Definition 


Anchored Scale 


Assigned Score 


2. ENVIRONMENT 
2.3. Informal Setting 


2.3.1. The consumer- 

physical 

Environment comfortable 

setting with spaces 
arranged to 
create a sense of 
safety, belonging, 
and support. 

■ Lack of physical comfort in the consumer-operated service that - 

is perceived by a sizeable minority of participants as intolerable 

or as extremely objectionable. 

■ Shortcomings in physical comfort are significant in the consumer- 
operated service, but would rarely be considered intolerable. 

■ The consumer-operated service offers settings in which the majority 
of members feel physically comfortable, even though there is obvious 
room for improvement. 

■ The consumer-operated service not only meets all obvious requirements 
for physical comfort, but makes efforts to ensure that even minor aspects 
of the environment add to member comfort. 

2.3.2. Rigid distinctions 

between members 

Environment and staff such as 

"provider" and 
"client" do not exist. 

■ Obvious devaluation of members with differentiation between members - 

and staff, such as strict separation of staff and member areas. 

■ Minor distinctions exist, such as that members may have to knock on 
staff doors, but not vice versa. 

while some program 
components may 
be structured, there 
remains a sense 
of freedom and 
self-expression. 

■ Staff attitudes are somewhat cold and distant, with minor distinctions 
between staff and members apparent. 

■ Staff treats members with openness, directness, and sincerity. 

Minor distinctions between staff and members are present. 

■ Staff/member interaction appears near ideal with openness, 
directness, and sincerity. No sense of inequality or staff/member 
distinctions are apparent. 

2.3. Informal Setting 


2.3.3. The Consumer- 

Sense of operated service 

Community provides a sense of 

fellowship in which 
people care about 
each other and create 
community together. 

■ Formal relationships among members, but little opportunity 

to informally relate with others or develop a sense of belonging. 

■ Formal relationships among members, but some opportunity 

to informally relate with others or develop a sense of belonging. 

■ Both formal and informal relationships among members with 
considerable opportunities for participants to informally relate with 
others or develop a sense of belonging. 

■ General comfort among participants characterized by extensive 
opportunities for warm, interpersonal interactions, sense of belonging, 
and numerous opportunities to socialize with other consumer-operated 
service participants. 


2.4. Reasonable Accommodations 

NOTE: Additional items about reasonable accommodations are under "Accessibility” (section 2.1.5 


2.4.1. No pressure to 

Timeframes J°'" “nsumer- 

operated service 
and no time limit 
for participation. 
Schedules are flexible 
and adapted to 
individual needs. 

■ Strict limitation of tenure in program; no opportunity for flexibility 
based on individual need. 

■ Some time limits, but also some flexibility based on individual need. 

■ No formal time limits, with some expectation of continued participation 
based on individual need. 

■ No time limits, with expectations about continued participation based 
on individual need. 
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Ingredient 


Definition 


Anchored Scale 


Assigned Score 



3.1. 

Peer Principle 


Relationships are 
based upon shared 
experiences and 
values. Staff and 
members share 
their experiences 
of having psychiatric 
disabilities. 

The relationships 
are characterized 
by reciprocity 
and mutuality. 

A peer relationship 
implies equality, 
along with mutual 
acceptance and 
mutual respect. 


Self-disclosure is limited, and no staff/leaders are identified as mental 
health consumers. Staff/leaders who are mental health consumers do 
not reveal this to consumer-operated service participants. 

Some self-disclosure by program staff/leaders, but this is limited to one 
or a few instances. 

Self-disclosure is common, but not universal among staff/leaders 
and participants within the consumer-operated service. 

Self-disclosure is almost universal—both participants and staff/leaders 
characterize their relationships as mutual/reciprocal. 


3.2. 

Helper's 

Principle 


Helping oneself and 
others is a corollary 
of the Peer Principle. 
Working for recovery 
of others facilitates 
personal recovery. 
Help or advice is 
friendly rather than 
professional and 
does not demand 
compliance. All 
services within the 
consumer-operated 
program are based 
on peer-to-peer 
relationships. 


No consumer-operated service participants report the experience 
of helping others. 

A few participants report some experience helping other 
program participants. 

Some participants report the experience of helping other participants. 
Most participants report the experience of helping other participants. 


3.3. Empowerment 

3.3.1. 

Personal 

Empowerment 


A sense of personal 
strength and efficacy, 
with self-direction 
and control over 
one's life. 


No members agree that being involved with the consumer-operated 
service has helped them make positive changes in their lives. 

Some members agree that being involved with the consumer-operated 
service has helped them make positive changes in their lives. 

About half the members agree that being involved with the consumer- 
operated service has helped them make positive changes in their lives. 

Most members agree that being involved with the consumer-operated 
service has helped them make positive changes in their lives. 

All members agree that being involved with the consumer-operated 
service has helped them make positive changes in their lives. 


3.3.2. 

Personal 

Accountability 


Consumers are 
expected, but 
not forced, to be 
accountable for 
their actions and 
to act responsibly. 
Self-reliance is 
encouraged. 


Consumer-operated service staff/leaders are often patronizing, placing 
few or no demands on participants. 

Consumer-operated service staff/leaders are somewhat patronizing, 
placing few or no demands on participants. 

Consumer-operated service staff/leaders are rarely patronizing but place 
few demands on program participants. 

Consumer-operated service staff/leaders are never patronizing and place 
modest demands on participants. 

Consumer-operated service staff/leaders encourage a high level 
of participant accountability and self-reliance. 
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Ingredient Definition Anchored Scale 


3. BELIEF SYSTEMS 
3.3. Empowerment 


Assigned Score 


3.3.3. 

Group 

Empowerment 


Belonging to an 
organized group that 
is recognized by the 
larger community 
contributes to the 
empowerment of 
participants. 

Members take an 
active role in the 
governance and 
decisionmaking 
processes within 
the consumer- 
operated service. 


There is no participant recognition of belonging to a group or to the 
consumer-operated service. 

There is some participant recognition and feelings of membership 
in a group. 

There is significant participant recognition and feelings of membership 
in the group. Membership affords opportunity for participants to 
contribute to consumer-operated service activities and planning. 

There is high participant recognition and feelings of membership in 
the group. Membership affords great opportunity for participants to 
contribute to consumer-operated service activities and planning within 
and beyond the group. 


3.4. 

Choice 


Participation in the 
consumer-operated 
service is completely 
voluntary. Consumers 
are regarded as 
experts in defining 
their own experiences 
and in choosing 
services that best 
suit them. Choice of 
services includes the 
right to choose none. 


Limited choice is apparent to participants. Participation is involuntary. 
Members can choose to participate or not. 

Members have the choice to participate and the opportunity to choose 
between at least two activities. 

Members have the choice to participate and the opportunity to 
choose between at least two activities with different levels/forms 
of participation. 

Members have the choice to participate in a wide array of program 
activities with different levels/forms of participation, including the 
opportunity to shape the activity. 


3.5. 

Recovery 


A positive recovery 
process acknowledges 
strengths and 
enhances well-being. 
Recovery cannot 
occur without hope, 
which provides the 
person with all the 
essential elements of 
recovery: the courage 
to change, to try, and 
to trust. 

The recovery process 
is different for each 
individual and, 
therefore, it is never 
defined rigidly, or 
forced on others. 


There is little or no recognition of a need for a hope-oriented 
recovery approach in the mission statement or in materials describing 
the program. 

There is some recognition of a need for a hope-oriented recovery 
approach in the mission statement or in materials describing 
the program. 

The mission statement and materials describing the consumer-operated 
service include a clear statement of a hope-oriented recovery approach. 

Not only does the mission statement and materials describing the 
consumer-operated service include a clear statement of a hope-oriented 
recovery approach, but staff and members can articulate the approach. 
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Ingredient 


Definition 


Anchored Scale 


Assigned Score 


3. BELIEF SYSTEMS 


3.6. Spiritual beliefs, 

Spiritual Growth Practices, and 

subjective, 
transcendent 
experiences are 
respected as 
an aspect of an 
individual's search 
for meaning and 
purpose in life. 

Such beliefs are not 
labeled as symptoms 
of illness. However, a 
consumer-operated 
service may have 
restrictions about 
proselytizing for a 
particular religion 
during the hours 
of operation. 

■ Expressions of spirituality and/or explorations of meaning and - 

purpose are not allowed or are discouraged within the consumer- 

operated service. 

■ Expressions of spirituality and/or explorations of meaning and purpose 
are neither discouraged nor encouraged within the consumer- 
operated service. 

■ Expressions of spirituality and/or explorations of meaning and purpose 
are allowed within the program and consumers feel comfortable sharing 
their beliefs, but there is little opportunity for consumers to share 

their beliefs. 

■ Expressions of spirituality and/or explorations of meaning and purpose 
are respected within the program and consumers have opportunities to 
share their beliefs. 
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Ingredient 


Definition 


Anchored Scale 


Assigned Score 


4. PEER SUPPORT 

4.1. Peer Support 

4.1.1. 

Formal Peer 
Support 


Organized mutual 
support groups 
based on common 
experience in which 
peers are available 
to each other for 
empathy and to 
share experiences 
and information. 
Trained peers offer 
individual support 
and information 
as part of the 
formal consumer- 
operated service. 


No formal peer support is offered to consumer-operated 
service members. 

Some formal peer support groups are offered to consumer-operated 
service members, but opportunities for these groups are on an 
irregular basis. 

At least one formal peer support group is offered to consumer-operated 
service members on a regular basis. 

More than one formal peer support group is offered to consumer- 
operated service members on a regular basis. 

Numerous peer support groups and other peer support services are 
offered to consumer-operated service members on a regular basis. 


4.1.2. 

Informal Peer 
Support 


Mutual support 
is provided in 
unscheduled group 
interactions and 
within individual 
relationships. 


The consumer-operated service offers no opportunities for peers 
to provide support to each other on an informal basis. 

The consumer-operated service offers few opportunities for peers 
to provide support to each other on an informal basis. 

The consumer-operated service offers some opportunities for peers 
to provide support to each other on an informal basis. 

The consumer-operated service offers many opportunities for peers 
to provide support to each other on an informal basis, and actively 
engages in the development of strong mutual peer relationships. 


4.2. 

Telling 
Our Stories 


Sharing personal 
accounts of life 
experiences as 
a mental health 
consumer is a 
cornerstone of 
promoting peer well¬ 
being and recovery. 
Opportunities to tell 
one's story and open 
discussion about such 
stories are embedded 
in peer support 
groups, in peer-to- 
peer interactions, 
at public forums, 
and within boards 
and committees. 
Sharing life 
experiences may 
also be a tool for 
public education. 


Sharing stories is actively discouraged on the basis that the practice 
has negative effects on the individual or the groups. 

The consumer-operated service has some provisions for sharing of 
stories about one's personal experiences and beliefs. However, these 
opportunities are limited or superficial. 

The consumer-operated service limits telling stories to social situations. 

The consumer-operated service provides regular opportunities for 
sharing stories among program participants. 

The consumer-operated service provides numerous formal and informal 
opportunities for sharing stories within the program and to the 
larger community. 
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Assigned Score 



4. PEER SUPPORT 


4.2.1. 

Artistic 

Expression 


Artistic expression 
is seen as a vital 
component of the 
consumer-operated 
service. It is valued 
as a way to explore 
personal meaning, 
express and expand 
talents, facilitate 
empowerment, and 
educate others about 
mental illness. 

Consumer-operated 
program members 
have the time, 
space, materials, and 
assistance to express 
themselves through 
artistic endeavors. 


There is no provision or outlet for artistic expression. 

There are some provisions or outlets for artistic expression, but these 
are minor or rare. Few resources are available. 

There is one regular artistic outlet with adequate resources (such as 
an art class or ongoing newsletter). 

There are regular artistic outlets with adequate resources that provide 
multiple opportunities for artistic expression. 

There are regular artistic outlets with enhanced resources that provide 
multiple opportunities for artistic expression using a variety of media. 
Artwork is prominently exhibited at the consumer-operated service. 


4.3. 

Consciousness 

Raising 


Members learn 
about the consumer 
movement, are 
encouraged to look 
beyond themselves, 
to work together, to 
help fellow peers, 
and to contribute to 
a larger consumer 
community. 


Most members think of themselves as uniquely ill or malfunctioning, 
keep their illness a secret, and feel disconnected and ashamed of it. 

Some members think of themselves as uniquely ill or malfunctioning, 
keep their illness a secret, and feel disconnected and ashamed of it. 

Most members do not think of themselves as ill or malfunctioning. 
They feel comfortable in connecting to a larger consumer community, 
but may not feel confident in contributing to this community. 

Most members recognize themselves as a valuable part of a 
larger consumer community and feel confident contributing to 
this community. 


4.4. Crisis Prevention 


4.4.1. 

Formal Crisis 
Prevention 


Members and 
staff learn to 
recognize psychiatric 
problems and how 
to address them 
before they escalate. 
Symptoms are 
reduced, involuntary 
commitment 
minimized, and 
recovery is fostered 
through individual or 
group peer support. 


No formal provisions are made for crisis prevention. 

At least one peer practice or program is available for formal crisis 
prevention; however, it is inconsistently used. 

At least one consistently used peer practice or program is available 
for formal crisis prevention. 

Multiple peer practices and programs are consistently used for formal 
crisis prevention and these appear to be effective. 


4.4.2. 

Informal Crisis 
Prevention 


Organic or 
spontaneous mutual 
support serves as a 
means of averting 
crises outside of any 
formal framework. 


No informal provisions are made for crisis prevention. 

At least one peer practice is available for informal crisis prevention: 
however, it is inconsistently used. 

At least one consistently used peer practice is available for informal crisis 
prevention and appears to be effective in providing regular outreach 
to members. 

Multiple peer practices are consistently used for informal crisis 
prevention and these appear to be effective in providing regular 
outreach to members. 
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Anchored Scale 
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4. PEER SUPPORT 



4.5. 

Peer Mentoring 
and Teaching 


■ Few members of the consumer-operated service report that there - 

are others within the program that they look up to. 

■ Some members report that there are others within the program 
that they look up to. 



■ Most consumer-operated service members report that there are others 
within the program that they look up to. 



■ Almost all consumer-operated service members report that there are 
others within the program that they look up to, and from whom they 
can receive guidance, support, and companionship. These relationships 
occur without regard to title or position within the consumer- 
operated service. 
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5. EDUCATION 

5.1. Self Management/Problem-Solving Strategies 

5.1.1. 

Formally 
Structured 
Problem-Solving 
Activities 


Practical skills 
promote strategies 
related to personal 
issues, symptom 
management, and 
support needs. The 
focus is on every day, 
practical solutions to 
human concerns. 
Regular activities 
focus on the 
development 
of skills or the 
dissemination of 
needed information; 
these activities 
use structured 
formats and include 
specific objectives. 


■ No classes with structured curriculum designed to teach self 
management or problem solving are offered to consumer-operated 
service members. 

■ Occasional classes provided for a small proportion of participants 
(up to 24%) with no structured curriculum or 

Classes currently are under development but have not yet been offered or 

Staff is trained in problem solving and self-management, but there is little 

evidence of the practical use of these techniques. 

■ There is evidence that classes with a formal curriculum in problem 
solving and self-management are being offered, and a substantial 
minority (25-49%) of members have participated in classes with 
structured formats designed to teach self-management and problem¬ 
solving strategies. 

■ There is evidence that classes with a formal curriculum in problem¬ 
solving and self-management are being offered, and a majority 
(50-74%) of members have participated in classes with structured 
formats designed to teach self-management and problem¬ 
solving strategies. 

■ There is evidence that classes with a formal curriculum in problem 
solving and self-management are offered, and most or all consumer- 
operated program members (75- 100%) have participated in classes 
with structured formats designed to teach self-management and 
problem-solving strategies. 


5.1.2. 

Receiving 

Informal 

Problem-Solving 

Support 


Less than 20 percent of members report that they received informal 
support in self-management or problem-solving assistance. 

Unstructured, peer-to-peer exchange of personal, lived experience 
is encouraged to enhance individual problem-solving abilities. 

A significant minority of members (20- 39%) report that they 
have received informal support in self-management or problem¬ 
solving assistance. 

About half of the members (40-59%) report that they have received 
informal support in self-management or problem-solving assistance. 

A majority of members (60-79%) report that they have received 
informal support in self-management or problem-solving assistance. 

Most or all of the members (80- 100%) report that they have received 
informal support in self-management or problem-solving assistance. 


5.1.3. 

Providing 

Informal 

Problem-Solving 

Support 


Peers help each other 
solve problems on an 
ad hoc basis using 
skills that they have 
acquired through 
the consumer- 
operated program or 
independently. 


Less than 20 percent of the participants report that they have provided 
informal support in self-management or problem-solving assistance. 

A significant minority of participants (20-39%) report that they 
have provided informal support in self-management or problem¬ 
solving assistance. 

About half of participants (40-59%) report that they have provided 
informal support in self-management or problem-solving assistance. 

A majority of participants (60-79%) report that they have provided 
informal support in self-management or problem-solving assistance. 

Most or all of the participants (80- 100%) report that they have 
provided informal support in self-management or problem¬ 
solving assistance. 
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Ingredient 


Definition 


Anchored Scale 


Assigned Score 



6. ADVOCACY 


6 . 1 . 1 . 
Formal Self 
Advocacy 
Activities 


Consumer-operated 
program members 
learn to identify 
their own needs 
and to advocate 
for themselves 
when there are 
gaps in services. 
Consumer-operated 
program members 
learn to become 
active partners in 
developing their 
own service plans 
with traditional 
organizations, and to 
deal effectively with 
entitlement agencies 
and other services. 


There is no formal self advocacy training, or informal, peer-to-peer self 
advocacy support. 

Less than one quarter of the members have participated in formal self 
advocacy training, or informal, peer-to-peer self advocacy support. 

A minority (25-49%) of members have participated in formal self 
advocacy training, or informal peer-to-peer self advocacy support. 

A majority (50-74%) of members have participated in formal self 
advocacy training, or informal, peer-to-peer self advocacy support. 

Most (75- 100%) members have participated in formal self advocacy 
training, or informal, peer-to-peer self advocacy support. 


6.2 

Peer Advocacy 


Members of the 
consumer-operated 
program assist other 
peers in resolving 
problems they may 
encounter on a 
daily basis in the 
community, such 
as problems with 
treatment providers, 
community service 
agencies, family 
members, neighbors, 
landlords, or 
other peers. 


There is no evidence of peer advocacy. 

Peer advocacy is a rare event or occurred on a one-time basis. 

There is some evidence of peer advocacy that happens in relation 
to other activities. 

There is evidence of formal peer advocacy, primarily provided by staff 
at the consumer-operated program. 

Most members are involved in providing peer advocacy. All members 
consider themselves as peer advocates. 


6 . 2 . 1 . 

Outreach to 
Participants 


The consumer- 
operated program 
makes concerted 
efforts to keep 
members informed 
of current activities 
and opportunities 
within and outside 
the program. 


No evidence of outreach to members. 

Some, but rare evidence that the consumer-operated program informs 
members by using Internet, newsletters, flyers, brochures, or holding 
regional conferences. 

Some evidence that the consumer-operated program regularly informs 
members by using Internet, newsletters, flyers, brochures, or holding 
regional conferences. 

Most members are informed by the consumer-operated program 
through Internet, newsletters, flyers, brochures, or holding regional 
conferences. There is a regular and strong advocacy content in materials. 

All members are informed by the consumer-operated program through 
multiple channels such as the Internet, regular newsletters, flyers, 
brochures, or holding regional conferences. There is a regular and 
strong advocacy content in materials. 


Total 
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FACIT Scoresheet Totals 


Domain Rater Score 

Conciliated Score 

1. Structure 


2. Environment 


3. Belief Systems 


4. Peer Support 


5. Education 


6. Advocacy 


Total 
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EVIDENCE-BASED 

PRACTICES 

Kfr 


Knowledge Informing Transformation 
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This Excel Workbook User’s Guide was developed 
as a companion piece to Appendix A: FAC IT 
Protocol, a.nd Appendix B: FAC IT Scoresheet. 

This User’s Guide provides a brief overview of 
the development of the FAG IT; an Excel 2003 
Workbook (on the GD-ROM that is included 
as part of this KIT) developed for your consumer- 
operated service to chart the scores on each of the 
Gommon Ingredients Domains; and a User’s Guide 
to the features of the workbook. 


The content of this User's Guide 


■ About the COSP Multisite 
Research Initiative 

■ FACIT Excel Workbook 

■ Workbook Features 

■ National FACIT Benchmark 

■ Program Worksheet 

■ Entering Data 

■ Password Procedure 

■ Copying Charts 

■ Printing Charts 


About the COSP Multisite 
Research Initiative 

The Gonsumer-Operated Services Program 
(GOSP) Multisite Research Initiative was funded 
by the Substance Abuse and Mental Health 
Services Administration (SAMHSA), Genter for 
Mental Health Services (GMHS) from 1998 to 
2007. The research mandate was to explore the 
impact of consumer-operated services when used 
as an adjunct to traditional mental health services 
for adults with serious mental illness. 

The primary goals of the initiative were to— 

■ Examine tlie effect of consumer-operated 
services on empowerment, housing, employment, 
social inclusion, and satisfaction witli services and 

■ Determine how participation in consumer- 
operated services affects costs for inpatient 
hospitalization, crisis intervention, and 
emergency room utilization as well as how 
it offsets costs in housing, criminal justice, 
vocational rehabilitation, physical health care, 
and income support. 

There were also secondary goals for the study. 
These were to create strong, productive 
partnerships among consumers, service providers, 
and researchers, and disseminate the knowledge 
gained about the effectiveness of consumer- 
operated programs and the specific components 
that contribute to their success. 
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With more than 1,800 participants, the COSP 
Multisite Research Initiative was the largest and 
most rigorous study of consumer-operated services 
in history. Interviews were administered at baseline, 
4, 8, and 12 months to adult mental health 
consumers randomized to one of two groups: 

■ Using traditional mental health programs only or 

■ Using traditional mental health programs 
and consumer-operated services. 

Results of the study provided evidence that the 
well-being of people with mental health diagnoses 
is strengthened through participation in consumer- 
operated programs. 

To ensure that the well-being outcomes found in 
the study could be attributed to the peer practices 
of the participating consumer-operated services, it 
was also determined that fidelity to the consumer- 
operated service model needed to be measured. 

It is impossible to forge a causal link between peer 
practices and wellness outcomes without knowledge 
that peer support practices are being delivered. 


Fidehty measures provide an objective rating 
system to assess the extent that components of 
a program are faithfully implemented according 
to intended program model and belief system. 
Most important, by measuring both outcomes 
and fidelity best practices can be identified. 

The Fidelity Assessment/Common Ingredients 
Tool (FACIT) was developed to assess the 
presence and strength of these “common 
ingredients.” During an extensive collaborative 
process with consumer-operated service providers 
and researchers, a set of 46 common ingredients 
of consumer-operated programs was identified 
across six different domains. 

Following the procedures described in the FACIT 
Protocol (See Appendix A of this booklet), two 
members of a FACIT evaluation team conducted 
a formal site visit at the consumer-operated program. 

Using the FACIT Scoresheet (see Appendix B 
of this booklet), they scored the program for each 
anchored item representing tlie common ingredients. 
If there was a discrepancy between scores given by 
individual team members, a conciliatory process 
was used to resolve these differences. 


Consumer-Operated Services 
Common Ingredients Domains 


■ Structure 

■ Environment 

■ Belief Systems 

■ Peer Support 

■ Education 

■ Advocacy 
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FAC IT Excel Workbook 

The FACIT Excel Workbook that is on the CD- 
ROM that is included as part of this KIT was 
developed as a simple data entry system for the 
consumer-operated program’s scores on each of 
the 46 common ingredients items entered in the 
FACIT scoresheet. In addition, it can automatically 
produce charts based on the scores. 

The data are to be used by individuals charged 
with program quality improvement to determine 
the fidelity of a consumer-operated service to 
evidence-based peer practices, and to monitor 
any changes to the program over time. 

Scores for each of the domains and specific 
items can help a program identify its strengths 
and weaknesses in providing peer support. 

The scores can also point to particular areas 
where peer competency training is required, 
changes are needed to build program leadership 
and member input, or facilities and activities 
must be expanded to promote the recovery 
and well-being of its membership. 


During the first year that a consumer-operated 
service participates in a quality improvement 
program, the FACIT evaluation team assesses 
the organization to establish a baseline or starting 
point for quality improvement efforts to become 
an evidence-based peer practice. 

The conciliated score of the two team members 
is entered into the Excel Workbook worksheets. 

In subsequent years, the consumer-operated 
service continues to conduct yearly assessments 
that can be compared to a national benchmark 
and to the program’s individual baseline. 

The national FACIT benchmark represents 
the aggregated FACIT scores from the eight 
consumer-operated programs that were selected 
to participate in the COSP Multisite Research 
Initiative. Their aggregated FACIT scores are a 
useful standard from which to compare individual 
program performance because these organizations 
are geographically diverse, represent multiple 
program models, and met the federal criteria 
for a consumer-run program. 
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Workbook features 

The FACIT Excel Workbook is designed in Excel 
2003 and consists of three worksheets. A worksheet 
is the Excel document used to store and work with 
data, and is also called a spreadsheet. A worksheet 
consists of cells that are organized into columns 
and rows. A worksheet is always stored in a workbook. 
There are tabs at the bottom of the screen identifying 
the names of tlie different worksheets. 

Once you click on a tab, the name of the worksheet 
is highlighted, so you know which worksheet is 
currently opened. 

The FACIT Excel Workbook contains three 
worksheets: 

■ National benchmark; 

■ Name of the specific consumer-operated 
service; and 

■ The chart for the consumer-operated service 
(e.g., Hope Center Chart). 

See Figure 1 below for a screen shot example 
of the data workbook with the three tabs for 
each of the worksheets contained in your FACIT 
Excel workbook. 


The National FACIT Benchmark Worksheet 

The first worksheet. Benchmark, is to be used as 
a point of reference to evaluate the FACIT scores 
of a consumer-operated program. 

The Benchmark worksheet contains each of 
the six FACIT domains and has the 46 common 
ingredients items associated with each of these 
domains. There is a column called Range, which 
displays the range of scores for each item. There 
are anchored scores for each of the 46 items, and 
ranges of 1-4 or 1-5 for each item with “0” used 
in special cases. 

For example, under the Structure domain, the 
first item scores board participation. There are 
anchored scores from I to 5. The lowest score, 
which is I, indicates that no member of the board 
is self-identified as a consumer. The highest score, 
5, indicates that between 90 percent and 100 
percent of the board self-identifies as consumers. 

There is also a column. Average, that reports 
the average score of the eight sites included in the 
COSP Multisite Research Initiative. This number 
is the national FACIT benchmark and can be 
compared with your organization s score for each 
item in the six domains. 
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Figure 1: National Benchmark Worksheet Screenshot Showing the Three Tabs of the Workbook 
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The Consumer-Operated Service Worksheet 


Figure 2 below shows an example of consumer- 
operated service data in the Structure domain. 


The second worksheet has the name of your 
consumer-operated service on the tab. Once 
opened, you will see each of the six domains, 
with the items for each of the domains. 


Figure 2: Consumer-Operated Service Worksheet: Year 1 

Scores Entered in the Structure Domain 


Conciliated Score I I 


Yr 1 

Yr 2 Yr 3 

Benchmark 

1.STRUCTURE 

1.1 Consumer Operated 

1.1.1 Board Participation 

1 


4.75 

1.1.2 Consumer Staff 

3 


4.88 

1.1.3 Hiring Decisions 

2 


3.94 

1.1.4 Budget Control 

1 


3.78 

1.1.5 Volunteer Opportunities 

2 


4.93 

1.2 Participant Responsive 

1.2.1 Planning Input 

1 


4 

1.2.2 Satisfaction/Grievance Response 

2 


3.72 

1.3 Linkage to Other Supports 

1.3.1 Linkage with Traditional Mental Health Services 

5 


3.5 

1.3.2 Linkage to Other Consumer-Operated Services 

2 


2.69 

1.3.3 Linkage with Other Service Agencies 

2 


2.91 

Total Structure Score 

21 

0 0 

36.19 
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As you enter the data, you will see a chart that 
compares the score for each item to the national 
benchmark scores. The charts are next to each 
of the domains and will automatically reflect 
the number that is entered. Figure 3 provides 
an example of Year 1 Item scores on the Structure 
Domain compared to the national benchmark. 

Looking at Figure 3, it is plain to see the areas that 
are below the national benchmarks as well as those 
where the consumer-operated service is equal to or 
higher than the national average. 


For example, look at Board Participation and see 
how much lower it is than the national benchmark. 

Also, notice that in regard to the items Planning 
Input and Satisfaction with the Grievance Process, 
the national benchmark equals or exceeds the 
national average. The colors of the bars on the 
chart were chosen to clearly show differences 
when printed or copied in black and white. 


Figure 3: Year 1 Item Scores on the Structure Domain Compared to National Benchmark/ 
Structure Domain 



Year 1 
Benchmark 
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The Consumer-Operated Service 
Chart Worksheet 

The third tab of the workbook contains a chart that 
maps the program s total domain scores against the 
national benchmarks. This chart is antomatically 
prodnced based on the total scores that are 
calcnlated by the worksheet for each of the domains. 


Fignre 4 is an example of what this chart looks like. 
Notice that the chart does not provide information 
on all 46 individnal practices assessed by the FACIT, 
jnst the aggregate resnlts for the six common 
ingredient domains. Again, it is easy to see how the 
program compares with the national benchmarks 
for each domain. 


Figure 4: Comparison of Program Total Domain Scores with National Benchmark Scores/ 
Comparison of Domain Scores to National Benchmark 


01 

o 

u 
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Structure Environment Belief Peer 

Systems Support 


Education Advocacy 


Domain 


■ Year 2 

■ Benchmark 
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Entering data 

Someone responsible for quality improvement at 
the consumer-operated service will be responsible 
for entering the scores from the annual 
FAC IT administration. 

The only data that need to he entered 
are on the worksheet with the na me of your 
consumer-operated service name on the tab. 

Be careful to enter the scores in the correct cells, 
as there are some rows that do not contain data. 
The total scores and charts are automatically 
produced based on the data you enter. 

Be careful to enter the scores for each item on 
the proper line and cell of the worksheet. Titles 
are entered on the worksheet to correspond 
to the actual FACIT scoring sheet, so there are 
some cells that do not require numeric entry. 

The total score for each domain is automatically 
calculated, so it is important to enter the data into 
each cell. Copying and pasting numbers into cells 
or ranges of cells may make it impossible for the 
software program to correctly calculate the total. 


Data protections 

The Consumer-Operated Service Worksheet has 
been protected to prevent anyone from accidentally 
changing, moving, or deleting the data. This prevents 
accidental input, as an entry in these cells would 
erase the formulas that calculate the totals. The 
benchmark numbers and charts are also protected. 

To aid in data entry, those cells that do not require 
a numeric entry have been protected, and therefore 
will not allow entries. Any modifications to tlie 
worksheets will require the protection feature to 
be disabled. To disable the protection feature, please 
refer to the “Password Procedure Feature” below. 

The software program will block the entry of scores 
outside the range for any item. For example, the 
range is from 1 to 4 for the Structure domain item 
“Hiring Decisions.” If you try to enter a “5,” you 
will get an error message stating that the range is 
from I to 4. Delete the incorrect entry and enter 
the correct score for that item. 
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Password procedure 

To make any changes to the worksheet, please 
use the following steps: 

1. Open the workbook, and select the 
“Agency Name” worksheet tab. 

2. On the toolbar, select Tools 

I Protection I Unprotect Sheet. 

3. A password is required to remove the 
protection status. The password is “facit” 

and should be entered in all lower case letters. 

It is recommended that the protection status 
be reinstated. To accomplish this, please follow 
these steps: 

1. On the toolbar, select Tools 
I Protection I Protect Sheet. 

2. At this point, select a password of your choice. 

It is highly recommended to use password 
protection, so you can either reinstate the 
password of “facit” or choose a new password to 
be used by your agency. 

3. Confirm the password by reentering 
the password that you chose. 


Copying charts 

The password protection feature prevents 
accidental entries, but it also prevents copying 
charts for reporting purposes. Therefore, please 
use the following steps to copy a chart: 

1. Open the workbook and select 
the “Org Name” worksheet tab. 

2. On the toolbar, select Tools 

I Protection I Unprotect Sheet. 

3. Enter the password of “facit” 
or the one you have chosen. 


4. Click on the chart that you want to copy. 

5. On the toolbar, select Edit|Copy. 

6. Go to the document that you want to Paste 
the chart into, and select Edit|Paste. You can 
change the size of the chart once it is in 

the document. 

7. Once you have completed copying the charts 
you want, remember to turn the protection 

on by deselecting the chart (just click in a blank 
cell) selecting Tools|Protection!Protect Sheet. 

8. Enter the password of your choosing. 

9. Confirm the password by reentering 
the password that you chose. 


Printing charts 

Take the following steps to print a chart: 

1. Open the workbook and select the 
“Agency Name” worksheet tab. 

2. On the toolbar, select Tools 

I Protection I Unprotect Sheet. 

3. Enter the password “facit” or the one 
you have chosen. 

4. Click on the chart that you want to print. 

5. On the toolbar, select FilejPrint. In the print 
dialogue box, make sure the “Selected Chart” 
radio button is selected under the 

“Print What” section. 

6. Once you have completed printing the 
charts you want, remember to turn the 
protection on by deselecting the chart 
(just click in a blank cell) and selecting 
Tools I Protection I Protect Sheet. 

7. Enter the password of your choosing. 

8. Confirm the password by reentering 
the password that you chose. 
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